What are screeners, and why are they vital for experience research?

Whether it’s a card sort, an online survey, a moderated or unmoderated remote usability test—if the people you recruit for the test aren't your target customers, the test will not be helpful. 

That’s not to say that you should only recruit your actual customers for a study. But there are people you’ll want to filter in or out, depending on what you’re trying to achieve.

Let’s start with the basics.

What is a screener question?

Before running any kind of usability testing, you’ll want to find people to participate. People who will carry out the tasks you sent them.

A screener (or screener question) is an opportunity for you to have more control over who takes your test.

This will also help you filter out anybody who wouldn’t be the best fit. 

For instance, if you’re testing the navigation of a website whose sole audience is civil engineers, and you want to ensure all the relevant categories are represented, there’s no point in recruiting an audience that doesn't even know what a civil engineer does.

The point is that in the above example, you would want to screen for a certain level of knowledge of civil engineering terms.

What does a screener look like?

You’ll likely have two types of questions available:

•      Multiple choice (user chooses one correct answer)
•      Checkboxes (user chooses all options that apply)

Be careful when selecting the type of screener question you want to use and think about how people might want to answer it. If the participant is likely to give more than one answer, then give them that option.

Here’s an example of a screener question for an online car dealership:

Which of the following statements apply to you?

•      I bought a car online over a year ago. (Screenout)
•      I bought a car online within the last year. (Next)
•      I’ve not bought a car but am planning to in the next 6 months. (Next)
•      I never have and never will buy a car. (Screenout)
•      I prefer not to say. (Screenout)

This is created so as to not lead the user towards one answer over another.

Avoiding Yes/No questions

Try to ask open questions, as yes/no questions are commonly seen to be too leading. For example, if you're looking to recruit Ford Drivers, then rather than asking, "Are you a Ford driver?", it would be more effective to ask which brand of car they drive and provide a list of car brands to choose from. Check out some more examples of leading questions to make sure you know what to avoid.

Can personas or demographics help find the right participants?

Before setting these questions, you’ll want to clearly define who your ideal test participant is. And if your team or organization has already defined your business’s key personas, the hard work has already been done.

Personas are a way to help organizations understand their potential and existing audience in a more personal way. In essence, personas are detailed profiles of a particular audience member, who represents a distinct group of people who share similar behavior, attitudes, personalities and preferences of your product, but are the figurehead for a larger demographic.

You could therefore tailor your questions to whittle down potential participants to the most valuable people.

Tips for writing a screener question

• Be direct and clear. sometimes it is easier to give the user statements that they may identify with instead of asking them a question.
• Remember that you’re dealing with people. You may have a very corporate tone in your business but this can be alienating and confusing to your participants. Write questions using the language of the user not the business.
• Don’t make it easy. People naturally want to help so they will try and find the answer they think you want. That’s why you should make sure it’s not obvious which answer you’re looking for.
• When setting up questions, try to envision how a participant would approach the question and aim to give the target audience the best chance of entering the study. If you were looking for a ‘small business owner’, some participants may identify themselves as ‘self-employed’ rather than a ‘business owner’, especially if they were just starting up. So, it would be worthwhile to let these people in and not wrongly screen them out, therefore limiting your sample.
• People may attribute themselves to more than one answer. For example, if you were looking to identify those with a certain medical disorder, people may have more than one, so you have to account for this in the question.
• Avoid leading questions. As Becca Kennedy states in her blog on avoiding leading questions, “Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t coloured by your own hopes or expectations. Your job is to listen, and to be deliberate with your words and actions.”
• If doing competitor analysis, the question: “Do you work for a competitor company?” is pointless as a participant probably knows the correct answer and may well lie to get selected. UserFocus suggests avoiding this by simply asking an open question: “Where do you work?” or “Tell us about your job?”
• Avoid beginning a test using a difficult screener, with the intention of simplifying it if struggling in field. On paper this makes sense and allows you to identify an incidence rate (percentage of those eligible to take part in the study) while you monitor progress. However, the danger associated with this is that you generally can not invite participants back who originally were screened out. So, in this instance, once you have relaxed the criteria of your screener, you are now at a disadvantage as you have access to fewer participants to take part in the study.

Final takeaways

Getting the best fit for your research involves juggling between three critical variables: representativeness, time in field and cost.

If exact representation is critical for your research questions then it will take more time in the field and often cost more. Usually when running research that requires assessment of a user’s ability to make a decision on specific things, high representation is important.

If getting results quickly is most important, then relaxing the representation criteria and being willing to pay a little more will ensure the fastest possible turn arounds. Classic usability type of questions could fall into this category.