What is a think out loud (TOL) study?

What is a think-out-loud (TOL) study?

TOL is a study that is used to get verbal feedback from participants. Questions in a TOL test are crafted in a way to elicit verbal and observational insights from participants.

This method can be used from the very beginning with wireframes to the final iteration of the website, and indeed, TOL usability testing is recommended throughout the whole product lifecycle to ensure that your website/app being developed is user-centered.

In particular, TOL testing is important for when you need to know the HOW and WHY behind the WHAT. For example, when new changes have been or will be made to a live site consider conducting a TOL study in order to ascertain user comprehension, usability and preferences.

What are typical use cases for conducting TOL?

There are many times when hearing what users have to say as they experience a site, app or prototype can be useful. As such there are many use cases for when a TOL study might be what you’re looking for to complete your research goals, and we certainly see a broad range of use cases from our customers.

Here are just a few of the more common ones that we see:

• Testing a wireframe to see what users would expect to see in certain areas.
• Having users go from low to high fidelity prototypes to ensure that the flow, content and call to actions are within the users’ mental model.
• To identify usability issues that might be causing drops in conversion or traffic.
• A/B testing to see and hear which design users prefer and why.
• Testing out new features/functions that may have been added to the website on the live or test environments.
• Better understand if there is a lack of comprehension/understanding.
• Detect common usability issues e.g. navigation, comprehension, findability etc.
• Validation/insight on how new or improved features and functions of the website/prototype are being perceived/used.
• Insight into the user’s mental model/behavior (expectations).

How does a TOL study work?

Just as with most research projects, you first start by identifying a research goal and writing a hypothesis. Then you design a test that will hopefully prove or disprove your hypothesis by writing research questions that will get to the heart of your issue.

Where a TOL study differs the most is that your questions and tasks have to be written and crafted in such a way that you get your participants to channel their thoughts aloud. This can be more challenging than you might think, but with practice and a little sound advice from a company who knows a thing or thirty about TOLs and you’ll be swimming in actionable feedback in no time.

Practical advice for running a TOL study

We have quite a bit of experience getting companies up and running with TOL studies. Here are a few of the most practical and helpful pieces of advice we give customers:

• In general, people love to talk about themselves. Use this natural human characteristic to your advantage. Warm up with questions like: “Tell me what you’re passionate about, and why” to get someone talking and in the mood to be chatty.
• TOL can be captured through either unmoderated or moderated sessions but keep in mind that in an unmoderated setting you want to try and plan for different variables and outcomes and ask various likely followup questions to keep users talking.
• Try to avoid formal language like “What are your perceptions of this site element?” Whether there is a moderator or not remain approachable! Instead say: “What do you think about this part of the site titled XYZ? Remember, there are no right or wrong answers.”
• We’ve found the sweet spot in regards to overall length of TOL studies is about 15 minutes. People tend to get fatigued expressing their thoughts verbally after that time and you can see a degradation in the quality of responses, to try and limit the amount of navigation tasks to fit that mark.
• Finally, the beauty of qualitative research is that you can really pull out meaningful feedback participants share about your experience or product. 

Give participants the ‘OK’ to delve into their imagination, and speak from the heart.

When should you use TOL?

Here are a few essential facts to consider while deciding on whether or not a TOL study is the right approach for your research goals:

Pros:

• Collect insightful verbal feedback.
• Helps you understand user behavior/mental models – Gives you an insight on how participants perceive the content, what paths they take, their expectations and what aspects of the website/app they find confusing or lacking.
• You get quotes and video clips from participants that can help reinforce a point when presenting to stakeholders.

Cons:

• Long analysis time – It can be time consuming when analyzing each video, especially if videos are longer than 15-20 minutes.
• Writing a good test design takes time and trial & error, but is worth it since the better the test design is the better insights you get.

What results do you get?

You get videos with rich audio insights from participants that need to be analyzed. What we usually do in house on the professional services team, and what we recommend customers to do for their stakeholders, is to create a report that is supported using clips/participant quotes that shows the answers to the hypotheses and or research questions and trends identified.

On top of this you will also get task metrics, such as Success, Time on Task, Clicks and Page views. And then depending on what type of supporting questions you include in your study you may also get likert scale ratings, NPS, open-ended text boxes, etc.

Tips for analyzing your results

This is a big one! Trying to keep coherent notes while analyzing videos and trying to put the whole picture together can sometimes be challenging, which is why we recommend including two things when analyzing TOL responses:

Analysis Matrix. In order to prevent your notes from looking messy while analyzing TOL videos we recommend that you create an analysis matrix like the image below. This will help you track significant observations, quotes, video time frames and start to track trends.

Issue severity chart. Using a issue severity chart, such as Low, Medium and High, will help you prioritize and identify the most important issues (if conducting a usability test).