How to avoid 'leading questions' in UX tests

What are leading questions?

In social science research, a leading question encourages a person to give a particular answer. Leading questions are not ideal for research because they introduce bias and influence the way participants respond.

Leading questions are also relevant to courtroom eyewitness testimonies. For example, if an examiner asks a witness whether he was home on the night of the murder, that’s a leading question. The phrasing assumes a murder indeed took place, and leads the witness to answer in a way that directly relates to his home. 

Why do leading questions matter for UX testing?

Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t colored by your own hopes or expectations. Your job is to listen, and to be deliberate with your words and actions.

People often tell you things they think you want to hear. It’s driven by natural kindness. But if your in-depth interview questions lead participants to give inaccurate answers, you might miss important UX insights.

Imagine the following scenario. You meet a research participant, Sally, and introduce yourself as a UX researcher. You say, “Thank you so much, Sally, for agreeing to participate in this research. I’m so excited to get feedback on this app. The team has worked very hard on it, and we know you’re going to love it.” Then, imagine you conclude your research session by asking Sally if there was anything she did not like about the app. She answers, “No, not really!”

Notice a problem? Sally went into this UX test already thinking a few things. First, you’ve probably given her high expectations for your app, and second, she doesn’t want to disappoint you or hurt your feelings!

Although it’s great to be polite and personable, avoid introducing too much emotion and personal expectations. Leading questions, like the one that gave Sally a hunch that you wanted her to respond favorably, lead to responses that aren't particularly true or useful. When creating user research questions, strive to remain neutral and ask open questions.

Specific examples of leading questions

It’s easy enough to understand how leading questions are bad for UX tests, but leading questions can be tricky beasts. Here is a list of questions. Which ones do you think are leading?

1. Would you rather use the old version or this improved version of the website?
2. What did you expect would happen when you pressed that button?
3. How would you normally perform this task, without this tool?
4. Do you find this feature frustrating to use?
5. Would you use this product if it were in your house?
6. What are some pros and cons of these designs?

Now, let’s break them down.

1. Would you rather use the old version or this improved version of the website? This is a leading question. By calling one version “improved,” you are already tricking participants’ brains into thinking that one is better than the other. It’s probably better to just call them Version A and Version B, or something like that.
2. What did you expect would happen when you pressed that button? This is a good question, especially when the user has already pressed the button so you are not suggesting they do so. This question keys into the user’s mental model for how the product works.
3. How would you normally perform this task, without this tool? This is another perfectly good question. It’s actually a really useful one, too, because it gives you extra context and juicy information about the problems you are trying to solve with the product.
4. Do you find this feature frustrating to use? A bit leading. As a researcher, you can deduce that a feature was frustrating through your own data or by the user explicitly saying so. But by asking the question directly, users are likely to think too hard about it and you might not get genuine answers.
5. Would you use this product if it were in your house? This question is not awful, but it is leading. Most people answer “yes,” so it doesn’t give you a lot of useful information. Try to get at this answer a different way. People really aren’t good at guessing about the future, so focus on past experiences and pain points.
6. What are some pros and cons of these designs? Another fine question.

How can you avoid leading UX research participants?

1. Avoid direct questions with single-word answers

They may not always be leading, but avoid questions that can have single-word answers, like yes/no questions. Many participants give the shortest answer possible, especially for written surveys. Remember that you really want the juicy stuff: stories, explanations, and examples.

Consider how to word questions to get richer, non-leading feedback. Rephrase direct questions into open questions, like “Please describe an example of…” or “Tell me about a time you…” People are good at speaking on their own experiences, and not very good at elaborating on direct questions.

2. Practice and plan ahead

It takes practice for UX tests and interviews to come easily, and we all make mistakes along the way. But you can plan ahead to minimize these pitfalls. While prepping for user interviews, write out the questions you intend to ask beforehand, and group similar questions together. Get feedback from team members or other UXers. Just be flexible about when the questions might naturally come up during the tests.

If you’re writing a survey, have others look it over and double check your wording. And if you have the resources, consider a pilot test of the survey questions—give it out to a handful of people before deploying it to a large audience. Essentially, there’s no reason why you can’t usability test the test itself.

3. Be mindful of your nonverbal behavior

Finally, beyond your words, your physical behavior can also be leading. Be encouraging and grateful for user feedback, but avoid responding too positively when participants say things you are hoping to hear, and avoid getting defensive in the face of product criticisms. Your poker face should be fully engaged.

Neutral behavior is easier said than done, so be mindful. During UX tests, we've seen research observers smile and nod when a participant behaves as expected, or point to features the participant didn’t see, or intervene to explain things the participant finds confusing. That’s all well and good during normal human interactions, but destructive during a UX test.

Of course, once the research session is over and you’ve collected your unbiased insights, feel free to open up, shake the seriousness off, and become human again with your participants. We actually encourage that part!