How to avoid bias in usability testing

Biased opinions are hard to avoid.

If you run a remote UX test, you should observe the interactions and listen for where testers are articulating any problems they have with your site. Less interesting is whether they like the color you chose for their background– that's more likely to be their own bias.

Where does bias occur?

When testing, bias can occur for various individuals at different stages, manifesting in endlessly surprising ways.

Bias about the way someone thinks something should be done

Call it gut instinct, call it intuition, call it an inkling – either way, your own thoughts about the way something should be done are not necessarily the way other people think it should be done.

Bias from experience

If most of your favorite websites have the navigation on the left-hand side, that’s how you’ll prefer to see it on any new site you visit, and feel like it’s wrong if it’s on the right-hand side. This doesn’t mean you’re right and the other is wrong – it just means you have a different experience from someone else.

Bias from best practice guides, third-party experts, or expert review

One organization may claim they improved conversions by a large percentage by making a button green, but it doesn’t mean your website will experience the same results. It’s a different product with a different audience.

Similarly, if something is established as a "best practice" in one industry, it doesn’t mean it translates to another. Plus, best practices change all the time.

Expert reviews, where a panel of UX experts gather to assess your product, are also prone to their own whims, experience and bias.

Your own mental model of the way something should work

A mental model is a tangible description of the way someone thinks something should work in the real world. Here’s a diagram of a mental model from Boxes and Arrows representing how someone might get up in the morning…

Each of these behaviors within the model is individual to that person and won’t necessarily translate identically to someone else.

Personal dislike

You might just take a personal dislike to the product, the service, the brand, your client, or even the person testing. It's sometimes hard to say why personal dislike comes up, but sometimes something rubs a user the wrong way. They can't shake that feeling enough to be objective.

Why is bias an issue in UX testing?

The purpose of user testing is to observe real-life interactions between a human being and your product, be it physical or digital. But this won’t be a pure observation if the tester has preconceived notions of how things should be, either through their own bias or the bias you’ve projected onto them. Bias can cloud the results of your test.

This also means that the user – i.e. your audience and your customer – isn’t the central focus of the experience. If you’ve bought in to UX testing and believe it’s the right path to improving your experience, then you have to be careful of your own bias. 

Gut instinct is fine to a small degree, as is expert opinion and experience, however none of this can be backed up without  user research and testing.

How to avoid bias

Unmoderated user testing

Unmoderated testing takes away the influence of others and removes outside distractions. People using your product are in their own natural environment, on their own with their own personal device, and best of all – comfortable. Certainly more comfortable than being sat in a lab, across from a ‘company representative’, being scrutinized on every decision you make.

This also means that people are less likely to tell you what you want to hear if they can’t see your face. People are more comfortable being critical from behind the privacy of their own computer screen. It also means that any stakeholders, product managers, or investors are kept well away from the test itself.

Screening questions

Screener questions can sometimes help filter out people with a certain opinion or political persuasion if you feel there may be a conflict of interest. But bear in mind there’s no strict guarantee that they won’t lie to get through anyway.

You should also remember that their observations are as valid as anyone else’s. I may not agree with the way Amazon goes about its business, but I could never deny the quality of its user experience.

Avoid leading questions

As Becca Kennedy writes, “Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t colored by your own hopes or expectations. Your job is to listen.”

So it is important to avoid leading questions like, “Would you rather use the old version or an improved version of the website?” or “Do you find this feature frustrating to use?” – basically avoid anything direct that can easily be answered with a one word answer.

Keep your own bias in check

If you want your tests to be as useful as possible, keep things as neutral as possible. Be professional and be open and receptive. Prepare for the tests in advance to ensure that the questions you asked are not biased. The more objective you can be, the better insights you'll get.