How to run a true intent study

What is a true intent study?

A true intent study is a type of survey in which you intercept a live visitor and ask them questions once they're done browsing your site or app. True intent studies are mainly used to better understand who is visiting your app or site and if they're able to accomplish what they came to do.

True intent studies are also helpful for consistently capturing such usability metrics as Satisfaction, NPS, SUS, etc.

What are typical use-cases for true intent testing?

True intent studies are typically conducted during the optimization phase of a site or app, although they're sometimes done during the discovery phase as well. The most frequent use cases for true intent testing are:

• Better understanding why users are coming to the site
• Seeing if visitors are able to complete key tasks
• Wanting to know what needs to be improved as well as how to prioritize them

True intent studies allow researchers to test the waters to see if live users are having any issues, and learn why they’re coming to a website or app in the first place.

How do true intent studies work?

A true intent study typically has three parts:

First, live visitors are intercepted on your app or site and asked if they would participate in your study. If they agree to participate in the study, they are taken to an initial questionnaire where you ask them what their main reasons were for visiting your site or app that day.

Second, after they answer the “why are you here today?” question, they'll get some instructions telling them to minimize the survey window, go back to the website, do what they came to do, and come back once they’ve finished the visit.

Lastly, once they're done with the visit, they'll come back to the survey. Here is where we have open-ended questions to understand if they accomplished what they came to do. These question reveal if they had any kind of difficulty and where those difficulties were found. You can also add rating scale questions to capture metrics such as NPS, satisfaction, SUS, etc.

Once you start uncovering where problems are coming from by looking at the open-ended comments, you can conduct follow-up usability tests to dig into the origin of these problems.

Practical advice for running a true intent study

• A JS code needs to be placed on the page where you want to intercept the visitors.
• This code could be re-used for future studies, so if you want to run true intent studies periodically, placing the code once is enough.
• The ideal number of responses in a study is typically 100 per main journey, adding up to about 400 total.
• The survey should have a maximum of 10 questions since participants typically aren't incentivized.

What are the pros and cons of true intent studies?

Here are essential facts to consider while deciding on whether or not a true intent study is the right approach for your research goals.

Pros:

• If you have a good amount of traffic you can get a lot of results very quickly with a low cost per participant.
• You can periodically re-use your true intent studies to easily capture the status of your website/app both in the moment and longitudinally over time.
• Not sure where to start with your research? True intent studies are a good method to get an understanding of which problems your users are facing.

Cons:

• Participants self-validate if they have been successful or not.
• If you ask open-ended questions ("Which difficulties did you have?" for example) there can be quite a lot of open data to go through.
• If you want to intercept participants in pages with low traffic, it could take a long time to get the data you’re after.

What results do you get?

This is largely dependent upon the types of questions you're asking. Most often, however, we see:

• Percentages. For example, X% of people came to do XYZ; Success Rate, X% of people were able to do the thing they came to do.
• Rating scales with the different metrics you gathered. NPS, satisfaction, ease of use, etc.
• Open-ended responses. “I couldn’t do Y because…”

Tips for analyzing your results

You’re going to have a jumble of data in the beginning and that’s okay! To make the most sense of your data, we recommend filtering for the reason they decided to visit and looking at the success rate for that task individually. E.g. if multiple people said they came to sign up, take a look at the success rates for that.

Also, when looking at participants reasons for visiting, start looking at the problems/difficulties that the visitors are reporting and group them in themes. Is the problem in the search? Did they get stuck in the booking process? Are they reporting technical issues?

Keep in mind that if you’re a UserTesting customer, you can integrate your true intent study with your analytics tool and cross the data you're getting from UserTesting with your analytics data.