5 rules for creating a good research hypothesis

A hypothesis is a proposed explanation made on the basis of limited evidence. It is the starting point for further investigation of something that peaks your curiosity.

A good hypothesis is critical to creating a measurable study with successful outcomes. Without one, you’re stumbling through the fog and merely guessing which direction to travel in. It’s an especially critical step in A/B and Multivariate testing. 

Every user research study needs clear goals and objectives, and a hypothesis is essential for this to happen. Writing a good hypothesis looks like this:

1: Problem: Think about the problem you’re trying to solve and what you know about it.

2: Question: Consider which questions you want to answer. 

3: Hypothesis: Write your research hypothesis.

4: Goal: State one or two SMART goals for your project (specific, measurable, achievable, relevant, time-bound).

5: Objective: Draft a measurable objective that aligns directly with each goal.

In this article, we will focus on writing your hypothesis.

Five rules for a good hypothesis

1: A hypothesis is your best guess about what will happen. A good hypothesis says, "this change will result in this outcome. The "change" meaning a variation of an element. For example manipulating the label, color, text, etc. The "outcome" is the measure of success or the metric—such as click-through rate, conversion, etc.

2: Your hypothesis may be wrong—just learn from it. The initial hypothesis might be quite bold, such as “Variation B will result in 40% conversion over variation A”. If the conversion uptick is only 35% then your hypothesis is false. But you can still learn from it. 

3: It must be specific. Explicitly stating values are important. Be bold, but not unrealistic. You must believe that what you suggest is indeed possible. When possible, be specific and assign numeric values to your predictions.

4: It must be measurable. The hypothesis must lead to concrete success metrics for the key measure. If you choose to evaluate click through, then measure clicks. If looking for conversion, then measure conversion, even if on a subsequent page. If measuring both, state in the study design which is more important, click through or conversion.

5: It should be repeatable. With a good hypothesis you should be able to run multiple different experiments that test different variants. And when retesting these variants, you should get the same results. If you find that your results are inconsistent, then revaluate prior versions and try a different direction. 

How to structure your hypothesis

Any good hypothesis has two key parts, the variant and the result. 

First, state which variant will be affected. Only state one (A, B ,or C) or the recipe if multivariate (A & B). Be sure that you’ve recorded each version of variant testing in your documentation for clarity. Also, ensure to include detailed descriptions of flows or processes for the purpose of re-testing.

Next, state the expected outcome. “Variant B will result in a 40% higher rate of course completion.” After the hypothesis, be sure to specifically document the metric that will measure the result - in this case, completion. Leave no ambiguity in your metric. 

Remember, always use a "control" when testing. The control is a factor that will not change during testing. It will be used as a benchmark to compare the results of the variants. The control is generally the current design in use. 

A good hypothesis begins with data. Whether the data is from web analytics, user research, competitive analyses, or your gut, a hypothesis should start with data you want to better understand.

It should make sense, be easy to read without ambiguity, and be based on reality rather than pie-in-the-sky thinking or simply shooting for a company KPI or objectives and key results (OKR). 

The data that results from a hypothesis is incremental and yields small insights to be built over time. 

Hypothesis example

Imagine you are an eccomerce website trying to better understand your customer's journey. Based on data and insights gathered, you noticed that many website visitors are struggling to locate the checkout button at the end of their journey. You find that 30% of visitors abandon the site with items still in the cart. 

You are trying to understand whether changing the checkout icon on your site will increase checkout completion. 

The shopping bag icon is variant A, the shopping cart icon is variant B, and the checkmark is the control (the current icon you are using on your website). 

Hypothesis: The shopping cart icon (variant B) will increase checkout completion by 15%. 

After exposing users to 3 different versions of the site, with the 3 different checkout icons. The data shows... 

• 55% of visitors shown the checkmark (control), completed their checkout. 
• 70% of visitors shown the shopping bag icon (variant A), completed their checkout. 
• 73% of visitors shown the shopping cart icon (variant B), completed their checkout.

The results shows evidence that a change in the icon led to an increase in checkout completion. Now we can take these insights further with statistical testing to see if these differences are statistically significant. Variant B was greater than our control by 18%, but is that difference significant enough to completely abandon the checkmark? Variant A and B both showed an increase, but which is better between the two? This is the beginning of optimizing our site for a seamless customer journey.

Quick tips for creating a good hypothesis

• Keep it short—just one clear sentence
• State the variant you believe will “win” (include screenshots in your doc background)
• State the metric that will define your winner (a download, purchase, sign-up … )
• Avoid adding attitudinal metrics with words like “because” or “since” 
• Always use a control to measure against your variant