How to ensure quality unmoderated test data

Unmoderated research is a very powerful tool. You get speed, efficiency and quick insights that can accelerate your team and move your product forward.

However, one of the challenges of unmoderated research is that, when relying on people participating from remote locations, you remove some of your own quality control.

There are a few steps that need to be taken when conducting online user research to ensure your speed and efficiency is met with quality.

The importance of data cleaning

If working with a research partner, make sure that they manage the panel of testers strictly. They should ensure bots aren’t included and continuously review the panel’s quality and engagement with different studies.

Once you have that partnership, ensure you make data cleaning part of the process.

Data cleaning is the process of detecting and correcting corrupt, inaccurate or irrelevant records from your database.

Some respondents may still not provide quality responses when answering questions. That is on par for the course in online user research. Market research has been managing this balance for years. In fact, the industry norm is to expect that 10-15% of participants from a study will need to be removed. 

One thing to keep in mind when removing respondents is asking yourself, “Is their answer really low quality, or does it make sense in the scenario?” For example, if they respond, “I don’t know” to a question, that answer may make sense and be sufficient. While you may not like the response, if you remove them, you may bias the data. Also, try to look at their answers holistically, across different questions.

In the market research industry, it is actually a best practice to include an open-ended question in surveys, to be able to easily filter out low-quality respondents. Again, it’s all part of the process.

So it’s important to make data cleaning part of your process for any unmoderated research you conduct. If you’re conducting a think-out-loud study with 10 participants, you’ll likely need to remove one person due to them not talking or poor audio quality. If you run a survey with 100 participants, you’ll likely need to remove about 10 participants due to nonsensical open-ended commentary.

At what point do I clean the data?

You can clean your data at different intervals throughout the test, depending on your preference. You can clean the data when you have 25% of the completes you need, 50%, or once you have 100% of the completes. Identify any respondents that need to be removed, and exclude them from the study. 

Steps to ensure quality unmoderated tests

When conducting a research plan, try to follow the steps below to ensure your final results are actionable, based on quality data:

1. Design your study script
2. Program the study
3. Test through the study yourself and make updates
4. Test the study with one to two colleagues and make updates
5. Soft launch the study – This means take the study live with a few participants (3-5) to see if in-the-wild participants have any trouble completing the tasks based on confusing instructions, etc. Make any updates here and remove the original participants if need be.
6. Full launch the study – Send it out to the necessary number of participants needed for this study.
7. Clean the data 
8. Replace those participants – Take the study live again and get new participants in. Continue to repeat steps 7-8 until you have 100% of the participants you need.
9. Analyze the data!

So when you run your next unmoderated project, be sure to embed data cleaning into your process. This step will not only ensure you have high-quality data, but will also save you time in the long run and ensure you’re making decisions with confidence.