Using customer segmentation to get the right amount of feedback

With all the arguments about the ideal number of usability study participants, it’s enough to make your head spin! Because of this, we’re often asked about how many people to test with during studies---but there isn’t an answer that works across the board. Fortunately, there are ways to estimate how many users you should test with.

At UserTesting, we often run qualitative remote usability studies. Qualitative studies help find general usability problems and trends by watching participants use an interface. The goal is to understand WHY the participants are experiencing usability problems so that improvements can be made.

Before you begin running studies, we first suggest grouping your users into your customer segments. Segmenting your customers is important because it helps you organize and group your target audiences. It seems straightforward, but different customers have different needs. One group’s behavior, likes, or needs may be different than another group.

What is a customer segment?

Customer segments (also known as user groups, personas, or user profiles) are groups of people that share certain characteristics. Segmenting your customers means focusing in on the group that buys your goods or services, to understand:

• What’s important to them
• What they need or are looking for
• Their typical web behaviors
• How to design for them
• What makes them convert
• How to customize the experience specifically for them

There are many customer segmentation methods. For example, you could segment your customers by:

• Industry
• Demographics
• Experience-level
• Buying behavior
• Lifestyle
• Customer type
• Income
• Profession
• Technical expertise
• Domain knowledge
• Location
• Web usage

There are different ways to approach this, so for a more in-depth explanation, check out this post on segmentation tactics.

How many customer segments do you need to test?

You may argue that you just want to test your “target audience.” And while that’s all well and good, you’ll typically have more than one customer segment within that target audience.

For example, let’s say you are a B2B company that sells a SaaS marketing solution. You know that your target audience is marketers, but you’re not sure how to break down that group further. But there are ways! You could segment participants by:

• Their knowledge about your product/industry: No knowledge vs. Some knowledge vs. Expert knowledge
• Their role: The information-gatherer vs. The decision-maker vs. The buyer
• Their buying phase: Research phase vs. Purchase phase

So how many customer segments do you need to test? That’s specific to your company, but it’s worth taking the time to think about and organize your audience into smaller groups. By doing so, you’ll be able to better target and design for the needs of the specific people visiting your site or app.

How many participants do I need?

When running qualitative studies, five participants is the often the accepted number. While there are many schools of thought, Jeff Sauro claims that five users will find approximately 85% of problems in an interface, given that the probability that a user would encounter a problem is about 31%.

However, while five is the generally accepted number, there are some instances in which you could include more (or fewer) users.

When do you need more than five users?

There are some situations when you should test with more than the recommended five users. For example, if you’re running a quantitative study and want to:

• Compare designs and prove one is better
• Gather user opinion and preference versus behavior
• Develop statistical proof that X% of users will show the same behavior

When could you use fewer than five users?

Fewer than five users may be acceptable in the case that your participants will run into some of the same usability issues. This could happen when:

• You test with users that have overlapping demographics within different segments
• Every one of your users is completing the same tasks within all segments

Do I need participants for every customer segment?

Overall, when setting up your qualitative study, you first need to break down your audience into specific customer segments. This will allow you to target and design for people visiting your site or app.

Once you’ve broken down your audience into customer segments, we suggest including 4-6 users in each group. By testing with multiple participants in each segment, you’ll find out how to give your customers a better experience, which will lead to better results for you!