Modern UX research in action: 10 research stories

The customer journey is often more complex than simply visiting a website. When most people think of remote usability studies, they visualize testing the user experience on websites and apps with participants sitting at home with their devices or in front of a computer.  While it’s important to provide an awesome online experience, the customer journey is often more complex than simply visiting a website for 15 minutes and making a purchase or exploring a new app for the first time.

These days, there’s a lot more you can do to uncover insights about how your target customers interact with your brand. With modern UX research tools, it’s possible to test customer experiences across all channels, from websites and mobile apps to in-person experiences and even physical products. For example, UserTesting’s mobile recorder allows you to record experiences beyond the device, so you can observe the participants’ surroundings and watch them interact with items in the real world. You can also study customer experiences in specific locations, like retail stores, restaurants, and public transit.

Plus, by experimenting with different methodologies, you can observe how people interact with your product over time, how different audience segments perceive your brand differently, and how you measure up to the competition in the eyes of your target market.

At UserTesting, we’ve set out to push the limits of our own platform, craft some unusual studies, and gather insights beyond traditional website usability testing.

Yelp’s mobile app destination study

Yelp’s mobile app is a bit of an enigma. It’s beloved by many, and it has tens of millions of downloads in the App Store and the Play Store... as well as thousands of 1-star reviews. However, reading through the one-star reviews, it’s clear that many are reviews for businesses that users must’ve accidentally submitted via Yelp’s in-app “Leave a review about this app” messaging. Oops!

The reviews of the Yelp app in the App Store don’t tell the full story about the user experience on the app itself. So the UserTesting team decided to conduct a study to explore how people use the app out in the real world.

Rather than doing a traditional usability study, we usedUserTesting’s mobile recorder to launch a destination study. Some folks hit the street looking for a restaurant to grab a bite to eat.

With their smartphone in hand, we asked our participants to tell us a bit about how they typically use Yelp, and then we had them use the app to find a lunch spot nearby.

Super Duper Burger has breakfast? I've never been there before.

What we learned

Participants liked Yelp because it offered all the important information they would need to choose a restaurant or other local business: hours, ratings, phone numbers, a map feature, etc.

The users felt like the pictures and reviews on Yelp helped to set realistic expectations of what they would actually encounter when they arrived at the restaurant in person.

Because there is so much information available on Yelp, our participants pointed out that having filters is a necessity—but the filters can get a little overwhelming and confusing.

Wrap-up

Because we ran this study with users walking around on the street, it felt like we were walking alongside our users as they took action. We got realistic feedback on how users interact with the app’s features in a distracting setting like a busy street while they’re hungry. We also got to see users interacting with the map and directions in real time.

Try a template

 If you're feeling inspired, you can get understand how customers engage with your brand or products across multiple touchpoints with this multichannel journey template. 

Fitbit device and app longitudinal study

Wearable technology for health and fitness has been on the rise for a while, and Fitbit’s products are some of the most popular.

We wondered: Does a band around your wrist really motivate you to exercise? How drastic is the impact on behavior changes? What causes people to keep wearing this device?

We ran a longitudinal study with five users and measured how (if at all) user behavior changed while wearing the Fitbit. During the first part of the study, we asked users to demonstrate how they interact with their Fitbit and the accompanying app. In the second part of the study, we checked in a week later to find out how often they had used their Fitbit in the last week and what, if anything, had changed in their behavior or habits since wearing the Fitbit.

“It makes me want to get up and move around when I have it on.”

What we learned

After one week of using the Fitbit, all five users stated they had noticed a change in their behavior and habits since using the Fitbit.

These changes included users stating they felt more inclined to stay active, especially in terms of how much they walked, and they felt motivated to keep track of what they ate throughout the day.

It makes me want to get up and move around when I have it on.

However, there was still room for improvement. Users mentioned some of the app’s features could have been explained more thoroughly, and several users wanted a tutorial to help them set the device up and become familiar with the functions.

Wrap-up

For the users in our study, a simple wristband really did impact how much they decided to exercise each day. The constant feedback and accountability users got from Fitbit helped them stick to their goals longer and made them more aware of their eating and exercising habits. It’s not just the device that leads people to have a healthier life; it’s the fact that Fitbit alerts people to lead a healthier life and gives them the awareness to change.

Try a template

Feeling inspired? You can assess critical app characteristics like appearance, ease of use, and trust with this mobile app evaluation template.

Nest Learning Thermostat unboxing

Watching customers unbox, assemble, and install your physical product can be as insightful as following what people are doing on your website. We ran an unboxing study with users who recently purchased the Nest Learning Thermostat to see how the product compared with their expectations.

Unlike ordinary usability studies that involve digital properties like websites and apps, for this study, we were only interested in the real-world experience with the product, so we launched a beyond-the-device study. Participants recorded their unboxing experience via their smartphone camera and UserTesting’s mobile recorder. They spoke their thoughts aloud as they opened the box, assembled the thermostat, and installed it in their home.

There’s a level on it too, right on the device, right? That’s awesome.

What we learned

Our participants found several unexpected delights as they installed their devices. One customer was happy the device had a back plate because it wouldn’t damage the paint on the wall. Another was pleased that it came with a built-in level, making installation a snap.

It’s valuable to see how customers discover your product’s features in real-time to catch their initial impressions, rather than waiting for them to write a review or complete a follow-up survey.

Wrap-up

Having users record the whole process of unboxing lets you get a peek into the world of first impressions and “fresh eyes.” It’ll help you better understand what delights and frustrates users, and you’ll figure out how to improve different aspects of your product.

Try a template

Feeling inspired? You can gain a deeper understanding of a customer's environment—and how your product fits into it—with this customer environment and context template.

Apple Watch exploratory study

The Apple Watch was a highly-anticipated product release. We wondered if it lived up to the hype. Would it make users’ lives easier? Which parts of the user experience would be awesome, and which could use some improvement?

We ran a qualitative, remote usability study with seven early adopters. Our participants ranged in age from 18 to 54 and were located in the US and the UK.

What we learned

Most participants agreed that the Apple Watch takes a little while to get used to since the interface is so different from other products. Some users said they wished the Apple Watch had certain features, like the ability to adjust the screen brightness and timeout, that it actually does have; they just weren’t immediately apparent.

Three of our participants didn’t receive notifications on the watch when they got a text message, and two weren’t sure whether setting “Do Not Disturb” on their iPhone would affect their Apple Watch, too. (It turns out that it will if you select the “Mirror iPhone” setting.)

“Controlling my iPhone (and what songs are playing) through my wrist is really cool and useful.”

Still, participants agreed that the Apple Watch did a great job with “little things that make life easier.” Sending a text was easy, thanks to good voice-to-text recognition, emojis, and pre-programmed replies like “Hey” and “On my way.” One user liked the ability to control the music playing from his iPhone through his Apple Watch, especially if his iPhone is in his pocket or plugged into speakers across the room. Several users noted they didn’t have to look at their phones as often, so they wouldn’t be distracted by other apps every time they wanted to read a notification.

Wrap-up

Despite the initial learning curve—and the fact that many third-party apps weren’t up to snuff yet—all seven of our participants said they would use the Apple watch on a daily basis, and they’d be very likely to recommend it to a friend. And, of the five participants who had used other smartwatches before, all five preferred the Apple Watch.

Snapchat demographic comparison

If you’re over 25, you may find Snapchat to be a bit of a mystery. It has a reputation as the app that teenagers use to share self-destructing pictures in a place where parents won’t find them. Its interface seems designed for a crowd that has grown up using mobile devices and apps rather than the desktop web. Plus, its website, billboards, and marketing materials tend to lack a clear explanation of why you should use it.

Here at UserTesting, this made us wonder how users of different age groups interact with the app for the first time. Do they get it? Do they like it? Do they feel like it was made for them?

We ran a remote usability study with 18 first-time Snapchat users: six who were between the ages of 18 and 24, six who were 25-40, and six who were over 40. We wanted to find out how the different age groups explored and got the hang of the app—and whether or not they felt like the app was made for them.

I don’t know what the number 1 means... what the ghost means... what the rectangular smiley face means.

What we learned

When our users opened the app for the first time, they arrived straight on the camera screen, and they saw their own face via the front-facing camera. (One groaned, “The first thing I see is myself with a double chin.”) A few wished there had been a tutorial to help them get started. Apparently, there is a video tutorial, but only three participants discovered it, and one didn’t understand how to play it.

Older participants acted more irritated than their younger counterparts by the lack of instructions and labels on the app’s icons. By contrast, the youngest age group was more confident poking around the app to discover each feature and the meanings behind the icons.

Even though the under-25 age group had the easiest time navigating Snapchat, a few of them still weren’t convinced that they needed it, noting that they could already share pictures using other apps. Older users were less likely to want to continue using the app, and several said it seemed like it was for “kids these days” instead of users their own age.

Wrap-up

Of the users who did enjoy Snapchat and wanted to keep using it, the primary reason was that they found it fun. None of the users described the app using words like “useful” or “helpful.” But Snapchat’s millions of users prove that sometimes, having a point isn’t the point. It’s about providing a fun and addictive service that resonates with just the right audience.

Google Wallet destination study

In a world of smartphones, smartwatches, and the Internet of Things, credit cards may soon be a thing of the past. Mobile payment apps like Google Wallet store users’ debit, credit, and gift card information, allowing them to make purchases online and in stores that use NFC—without the need for a plastic card.

We decided to explore how first-time users interact with Google Wallet. How do they feel about keeping their payment information in their phones? What’s it like to make a purchase in a store using only the Google Wallet app?

To answer these questions, we ran a destination study with five users. We asked them to install and explore the app on their phones, and we gave them each a $10 Walgreens gift card to load into the app. Then we asked them to go to their nearest Walgreens and buy something under $10 using the app.

I don’t know if you’re able to scan the barcode directly. You may have to type it in.

What we learned

In general, users were excited about the app. It was easy to set up, the interface was intuitive, and our participants immediately understood the navigation.

While Google Wallet requires users to select and enter a PIN to access the app, a couple of our participants wanted more reassurance that their information would be safe.

Making a purchase in a brick-and-mortar store was the most confusing part of the experience. Two participants said they wished they’d had clearer instructions while making an in-store payment for the first time. Three users noted that the store employees weren’t familiar with Google Wallet, so they weren’t able to offer any additional help to the users.

Wrap-up

Mobile payment apps like Google Wallet face the challenge of balancing convenience and security. While some users are quick to adopt technology that makes their lives easier and eliminates the need for repetitive tasks, others prefer to wait until more people they know have tried it and found it to be safe. Most folks aren’t throwing away their plastic cards just yet. But, if Google Wallet and its competitors continue to focus on providing a secure, easy-to-understand, and delightful user experience, we may start to use mobile payment apps more often than credit cards in the very near future.

Virgin America competitor study

Many travel companies have a strikingly similar user experience. When users book a flight online, they tend to expect a website that looks more or less the same as all of the other travel companies.

Virgin America has turned those expectations upside down in the hopes of creating a distinctive, memorable, and hopefully excellent customer experience. Everything from their ads to their in-flight safety messaging goes against the traditional industry norms.

But is it effective? Do users understand how to choose a flight? Do they find the workflow helpful or confusing? We wanted to see if Virgin America’s design choices create a delightful user experience—or just disorient new customers.

We ran a direct comparison study with six first-time Virgin America users. We asked each of them to select and book a flight from Chicago to San Francisco on VirginAmerica.com, then select and book a similar flight using a different website of their choice. We measured how easy or difficult they found the flight booking process on each website, and tracked the time it took them to select a flight on each website. Finally, we asked them to verbally compare their experiences on the two websites.

I’m guessing I have to click on the Main Cabin. There we go; now we’re moving.

What we learned

Users noticed right away that the experience on Virgin America was different. While some features were confusing at first (such as selecting departure dates on the calendar and picking a flight out of a list of options), the users reported that the process felt streamlined and enjoyable. All six participants were able to successfully select and book the flight. It took users a longer time on average to select a flight on Virgin America than on the other websites, but users still thought it was easy to use.

Wrap-up

Conventional wisdom tells companies to find out their users’ expectations and then match them. But many customers are more open-minded about online experiences than companies expect. Brilliant visual design, straightforward copy, and an intuitive interface can go a long way in helping users adapt to a new way of doing things. And, as we saw with our study of Virgin America, users are willing to excuse some hiccups in their workflow if the design appeals to their emotions and makes them feel delighted.

Netflix redesign usability study

With over 62 million users worldwide and 10 billion hours of movies and shows streamed in Q1 2015, it’s clear that Netflix has been doing something right. That’s why their website redesign in Q2 2015 could be considered a risky move. By changing something so popular, Netflix ran the risk of confusing and alienating existing users who loved the old design. It was critical for Netflix to keep their existing customers happy while also attracting new ones.

We recruited five Netflix subscribers to participate in a remote usability study. We screened for people who primarily watch Netflix on their desktop computers.

What we learned

Netflix’s new website was designed to be more consistent with its mobile, console, and smart TV apps. This creates a more unified omnichannel experience for customers who use Netflix on different devices. Most of our participants said the new design felt newer, sleeker, and more up-to-date.

Knowing where I am in this list is great. It would be better if I could click on it and jup to the end.

In the new design, users can hover over an image to expand a slideshow of select clips, along with more information on the movie. While users liked the idea of being able to learn more about a movie at a glance, the interface may take some getting used to. Four participants accidentally started playing the movie when they were attempting to find more info. Others lost their place when they accidentally hovered over a different movie while viewing details of one.

All five of our participants pointed out that they were glad the redesign still offered a great browsing experience, and they found it very easy to find movies that matched their personal taste preferences. Still, they agreed that there was one area that had room for improvement: the “My List” feature could be a little overwhelming, and several users had trouble locating it.

Wrap-up

With more and more people interacting with companies on multiple devices, it’s more important than ever to create a consistent experience across all channels. While no design is perfect, users appreciate knowing what to expect from a company. The “modern, sleek” feel that users noticed in Netflix’s new site may be doing just that.

Starbucks app destination study

The Starbucks mobile app introduced an exciting new feature in Fall of 2015: the ability to order and pay for drinks ahead of time, skipping the line in the store. While this seems like a huge convenience for many coffee drinkers, we wondered: Will Starbucks customers understand how to use the feature right away? How smoothly will the process of ordering, paying, and picking up go in the real world? What will people like and dislike about the experience?

We recruited six Starbucks mobile app users from around the United States for a two-part study. The first part of the study was a remote user interview. We asked participants how they typically use the Starbucks app, when and where they use it, and what value they find in it. The second part of the study was a destination study in which participants ordered a drink via the app and picked it up at a Starbucks location.

I don’t see my drink anywhere. Maybe I should talk to the cashier?

What we learned

In the interviews, we found that the two main reasons why participants used the app were status and convenience. Participants said they felt like VIPs when they used the app to gain rewards and load up their Starbucks cards. They also felt like it was convenient to be able to keep their Starbucks card on their phone, where it was always handy. In the destination study, we found that while it was easy to locate the nearest Starbucks, actually placing the order presented more challenges. Some users weren’t sure whether they needed to pick a location or a drink first. Two encountered errors stating their order could not be placed, but no explanation of why. Picking up the drink was the most confusing part of the process: some participants understood where to pick up the drink, others didn’t, one participant’s drink wasn’t ready when she arrived, and one had to start the process all over by getting in line to order her drink.

Wrap-up

Overall, participants had positive experiences, although no order and pick-up was seamless. Participants were excited about the idea to use the app to enjoy their favorite drink, even if it took a little longer than expected. Even though the ordering process wasn’t perfect, all the participants got what they wanted in the end—their favorite Starbucks fix! When technology didn’t quite work as planned, great service and loyalty to the brand stepped in to compensate.

iPhone 3D Touch exploratory study

With the introduction of 3D Touch for iPhone 6s and 6s Plus, the iPhone screen acts like a pressure-sensitive surface, making way for an additional “dimension” of interaction beyond swiping, pinching, and tapping. Users can now apply different amounts of pressure to initiate different types of interactions on their phone.

We invited six iPhone 6s users and two iPhone 6s Plus users to explore the feature in a remote usability study. We asked them to try out the Quick Actions on the home screen, the “Peek” and “Pop” gestures in a dummy email inbox, and the keyboard-turned-trackpad in Notes. All participants had already played around with their new phones, so they all had some degree of familiarity with 3D Touch or “Force Touch” prior to the study.

What we learned

Most participants admitted they were still getting used to the feature and felt they’d need to break their old habits from prior iPhones before they could take full advantage of the 3D Touch. The majority of participants encountered some trouble in applying the right amount of pressure. Two participants wanted to see some type of tutorial for 3D Touch to help them better understand how to correctly apply the different amounts of pressure.

If I press on it, there’s a little arrow up top there. If I swipe it, I can do these different tasks.

Even without mastery of 3D Touch, participants acknowledged that this new system allowed for smoother transitions from one interaction to another, and it saved them time by cutting out multiple steps in an action. One participant likened the 3D Touch to an advanced right-click, while several other participants admired the additional “layer” or “depth” of navigation that it allowed.

Wrap-up

More than anything else, the novelty of this technology stuck with participants as they worked to get used to the 3D Touch. When we asked participants to choose adjectives to describe the technology, a majority chose words like, “inventive,” “impressive,” “creative,” and “fun.” Not only did these adjectives reflect an overall positive experience, but it emphasized how fresh the experience was for them.