Qualitative heatmaps: how to visualize the frequency of usability issues

In the last year we have undertaken several large scale benchmark studies, typically comparing four or more sites at a time. Quantitative metrics from multiple sites from these benchmarks (such as ‘time-on-task’) can be quickly and effectively visualized in a single bar graph. However, condensing qualitative data into one slide presents much more of a challenge.

So let us recommend an easier way to deliver UX insights: qualitative heatmaps.

But first let’s take a look at your other options, and why they present a challenge.

The challenges of traditional thematic analysis

The standard method for analyzing qualitative comments is to perform a "thematic analysis". This is where each open comment is coded into a theme.

In the example below we can see that the first user simply reports a technical issue, whilst the second user reported both a technical issue and stated they were unable to enter their details into the booking panel.

When complete, the column headings of your thematic analysis will show you the full range of usability issues that were encountered. At this point you may be tempted to just summarize these issues in a paragraph, however it's vital that you quantify how often each issue was encountered.

For example, if 25 participants encountered technical issues and 1 struggled to locate the search functionality, it would be misleading to list these two issues together in a manner that implies they occur with equal frequency. So how would you summarize the data?

Summarizing thematic analysis data

The next step is to turn your counts in to percentages. Let’s say 5 of the 100 participants who completed a task on "Amazing airlines" reported technical issues. Simply divide 5 (participants who reported a technical issue) by 100 (the total number of participants in that segment) to get a figure of 5%.

Then repeat this step for each issue encountered on each site.

Visualizing thematic analysis data

When visualizing the data from usability tests for a single site you can simply list out the issues from the highest to the lowest frequency. For example:

• 21% experienced technical issues.
• 15% struggled to enter details in to the booking panel.
• 10% struggled to locate the search functionality.
• 9% could not work out how to proceed to next page.
• 8% were unable to sort the flights by departure time.
• 8% struggled to understand content on pages.
• 6% were unable to enter their personal details in to the forms.
• 2% unable to navigate back to previous steps.

However, because these lists are usually quite long, the data from one site usually takes up an entire slide. In benchmarks this means having to flick between four or more slides to compare the issues encountered on each site.

Our initial solution was to display the data in a bar graph like the one shown below. This allows all data to a be displayed on one page and gives a visual marker of how frequently the issue occurs on each site. However, the downside is that when you have a wide range of issues, the width of each bar becomes microscopic.

Solution: qualitative heatmaps

Recently a benchmark study we ran had over 20 quantitative metrics that were being compared across four sites. we condense these findings in to one slide that would instantaneously highlight which sites performed better/worse and show if there was any particular area of the sites that needed improvement.

A suggestion called an "overview heatmap" was raised, which would look something like this:

In the example above, the "traffic light" color coding makes it instantly apparent that Amazing Airlines is the best performing site while Disastrous Airlines is the worst. And, when looking at the metrics for Disastrous Airlines, it would appear that a poor search functionality was driving this lackluster performance.

So much insight on just one slide.

We began to wonder, could this technique be used to summarize qualitative data too?

A few experiments produced the format below. It has the same layout as the quantitative overview heatmap—but instead of the traffic light color coding it uses shades of red to indicate whether an issue is a high or low frequency occurrence.

In the example above, the red shading makes it immediately apparent that technical issues are a high frequency occurrence across all sites.

In addition to this, the heatmaps highlight the fact that a high proportion of those using Better Airlines were unable to sort the flights by departure time, over a third of those using Crummy Airlines could not enter their details into the booking panel and a quarter of users on Disastrous Airlines had difficulty locating the search function.

While other issues were reported, the pale shading indicates that they occurred at a low frequency that didn't warrant concern.

Qualitative heatmaps are now standard format for presenting qualitative data in reports. They satisfy most peoples' desire for maximum insight/minimum slides.