Benchmarking UX: how to track improvements over time

A fellow UX professional recently told me a fun story: Her boss came to her desk and said, “I want us to add benchmarking to our testing strategy.” When she asked him why, he told her that he’d recently heard that a competitor was doing it and wanted to keep up with the times.

Benchmarking is a hot topic in the UX industry, and if done properly, it can provide valuable information about how your site or app is performing over time.

Unfortunately, benchmarking is often perceived as “just another” way to test a product's usability. The truth is that benchmarking is a commitment that requires all the thought and reflection that any good commitment entails. Stick to the guidelines below, and you’ll gain a rich collection of data that showcases your team’s achievements and pitfalls across the life of your site or app.

What is UX benchmarking?

Benchmarking is the process of testing a site or app’s progress over time. This can progress through different iterations of a prototype, versions of an application, or even across different sites: yours and your competitors.

What’s the benefit of UX benchmarking?

Many companies like to run benchmarking studies because they show actual data as it changes over time. It adds a quantitative dimension to the qualitative research you’re already doing, which can drive home your findings and add weight to your suggestions.

Benchmarking can also help you understand how later iterations of your product solve problems, as shown in the graph above. Navigation has steadily improved in each iteration—it even reached a perfect score for tablet users in the latest round! Without watching a single video, you can rest easier knowing that users are getting through your app without difficulty.

How do you benchmark experiences?

Set the plan

It starts with a conversation among your team. You need to agree on exactly what you're hoping to learn, because once you have decided on script for a benchmarking study, you shouldn’t make changes to it. (That could compromise the integrity of the research!) So ask yourselves:

• What are we trying to measure? The tutorial for new users? What keeps current users coming back for more, or what makes them leave? How trustworthy we appear? How difficult or easy it is to complete the basic tasks on our site?
• Can we commit to this on a regular basis? Do we have the budget? How often are we producing new versions? How often do we want to see how we stack up against our competitors?

Write the script

Once you know the big questions that your team wants to answer, then you need a script---a very specific kind of script.

1. The tasks have to be goal-oriented and as basic as possible.

• For ThinkGeek, “Find a product you’d like to buy.”
• For Evernote, “Create a new note.”
• For Tax Slayer, “Sign up for an account.”

2. The tasks also have to avoid using signposts or descriptive instructions because those elements might not be there in the next round of testing.

• “Click the blue + icon to subscribe to this post.” = BAD
• “Subscribe to this post.” = GOOD

3. Follow each task with evaluative questions, and keep a consistent pattern of tasks and questions. Using measurable questions like rating scale or multiple choice will give you the data you need to make those pretty charts!

• Task: “Create a new playlist with songs from 3 different artists.”
• Rating scale: “How difficult (1) or easy (5) was it to create a playlist?”
• Multiple choice: “Compared to other music apps, was it easier, harder, or about the same to create a playlist?”
  • Easier
  • Harder
  • About the Same
  • Other/None of the Above

Once you’ve got a script, it’s important to keep it as consistent as possible. Any shifts in the tasks or questions can compromise data and make it more difficult to compare results across different rounds of testing, so try not to tinker with it!

Pick your user demographics

Finally, it’s important to capture the same number and type of users.

If your budget allows, include more users than you would in a typical study. For ordinary usability tests, 5-7 users will normally give you plenty of qualitative insight. But because we’re looking more at numeric trends in a benchmarking study, you can bump up your total number of users to 10, 20, or even 50.

You don't need to test with the exact same people each time; it’s okay to get fresh eyes on each version of your product.

However, the total number of users and any distinguishing characteristics should remain the same. For example, if you tested with 20 college students the first time around, you'll need to test with 20 college students for the next round too.

How often should I be running benchmarking studies?

The frequency and spacing of new studies depends on a variety of things, including how quickly new designs are being produced and how much of a budget you have.

• If your product is an established one, a quarterly benchmarking study may be all you need to keep tabs on any performance issues.
• If you’re pumping out new iterations every week, you’ll want to be a bit more aggressive with your scheduling... and probably lighter on the total number of users.
• And in some cases, your product may have sporadic updates, in which case the benchmarking may also be sporadic.

The important thing about benchmarking is seeing the progress over time, so whatever you do, don’t run an initial benchmarking study and then never run another one. You’ll never know what benefit a benchmarking study can really be.

Get a holistic experience measurement with UserTesting's QXscore

Quickly identify customer signals

QXscore™, short for Quality Experience Score, is a metric developed by UserTesting that empowers digital leaders to measure the overall quality of a customer’s experience across various organizational touchpoints. 

Unlike traditional metrics, QXscore goes beyond quantitative data to incorporate qualitative insights, providing a comprehensive view of customer experience.

With QXscore, leaders get a single 100-point score, weighted across specific metrics that directly reflect how customers engage with an organization's products and experiences. With this high-level scorecard, leaders can quickly understand and act on what's working and what's not with their overall customer experience.