Testing mobile experiences

In today’s rapidly evolving and highly competitive landscape, even the most well-established and successful organizations have to perform at peak across all channels. We're in what Forrester Research calls “the age of the customer,” in which companies must develop a deep understanding of what their customers value most and deliver superior digital experiences—or risk becoming irrelevant.

These days, so much of people's lives are spent using mobile devices and consuming mobile experiences. In fact, reports show that consumers spend more than five hours a day on their smartphones, doing everything from reading emails to checking their bank accounts to watching videos. The smartphone has become the go-to device for a wide range of activities including shopping, banking, job search, and many others.

Why it's important to test mobile experiences

Research shows that 77% of consumers don’t use an app three days after download. And one in four people abandons an app after just one use. Providing a useful and delightful experience ensures you see positive returns from all the work you put into developing your app and all the resources you spent on customer acquisition.

Apps aren't the only mobile experiences that matter though. It's estimated that 85% of adults think that a company’s mobile web experience should be as good as or better than on desktop. If you count the moments when you're desperately relying on a mobile experience to come through for you—like when you’re checking your account balance using a banking app while standing in line to ensure you don’t overdraw with the purchase you’re about to make—then you probably understand. Ensure your web and app experiences on mobile live up to your customers' high expectations.

Whether web or app for mobile, it’s important that your experiences on mobile devices meet customer expectations.

How to use this guide

In this resource, we’ll share why it's important for teams to rely on user feedback from mobile testing to guide the conception, development, design, and release of mobile experiences, and provide tips on how to effectively gather remote user feedback, broken out into three sections, based on the type of mobile experience you want to test.

Testing mobile app experiences

According to an eMarketer study, 88% of smartphone usage is spent using apps. This widespread usage means there is ample opportunity for app developers to get in front of users—but also means there's a lot of competition for user loyalty. In fact, according to eMarketer, most deleted apps are removed within a week after the last use, indicating that apps must prove their value to users with every use, or risk obsolescence.

The solution to this challenge? Consistent testing throughout an app's development—and after launch.

Testing your mobile app throughout development

Testing throughout the development cycle may be something you've heard before, but what does that mean in practice? Here are a few examples of when UserTesting recommends testing an app, as well as some examples to illustrate the insights you can gather from this type of testing.

Prior to development

Before a single line of code is written, it's crucial to test the concepts, wireframes, and prototypes of your app, first. Testing before development saves valuable time and prevents costly rework.

Prior to release

Once you're confident that your app will meet the needs of your target market, be sure to test again just prior to its release. Testing at this stage enables you to catch last-minute issues and ensure a smooth, successful launch. There’s little room for error after your app has launched. But until that moment, look at every mistake as an opportunity to learn from your users, and to iterate and improve your app until you’ve got an experience your users love.

After release

Once your app is released out into the world you'll probably feel a sense of accomplishment and relief. But don't rest on those laurels for too long, there's still a lot you can learn post-launch, including many of the insights you probably gathered pre-launch, such as app download first impressions, competitive testing, ease of use, and general adoption. Remember, testing shouldn't stop once your app has launched, continuous insights after launch will help you continue to improve your app experiences.

Mobile app testing tips

Although testing a mobile app is similar to how you'd test other digital experiences, there are a few tips and best practices to keep in mind to ensure the best results.

• Be aware of differences in operating system experiences: If you intend to release your app to both Android and iOS users, for example, be sure that you're testing with both, throughout the development cycle. This is where testing the app store and onboarding experiences can be critical. While the app itself may be identical on both platforms, the store and download experience may be quite different, so it's important to understand both and improve and optimize so that each has a similar experience for users. Note: this may require you to conduct several batches of the same test, one for each operating system.
• Limit written response questions: Written response questions can be helpful when analyzing the results from your studies, however, it's not necessarily the best experience for participants—especially on mobile devices. Tapping out just a few words with fingertips can quickly become tiresome and may lead participants to fatigue early, so opt for verbal responses as much as possible to keep your audiences engaged and on-task.
• Keep tests short and to the point: This is a good practice for any type of test but is especially important with mobile app testing. People are notorious for having short attention spans with digital experiences, so help keep your test participants engaged by keeping tests to 15 minutes or less.  
• Provide dummy information (e.g., logins and passwords): To avoid inadvertently collecting personally identifiable information, provide test participants with fake information they can use in their tests, such as email address, password, name, and date of birth. It's helpful to remind participants to make note of this information and keep it handy during the test, as they'll likely be switching back and forth between screens and may not have access to the initial instructions throughout the test.

See mobile app testing in action with UserTesting

Remote video URL

Testing mobile web experiences

It's not just apps that capture the attention (and thumbs) of consumers. Mobile web traffic continues to increase, with traffic to mobile web sites increasing over 11% from 2019 to 2020, according to data from SimilarWeb.

With a smaller screen, and thus less real estate for content, the mobile experience is inherently different compared to desktop. No doubt everyone can think of a time when a website didn’t quite translate on mobile, resulting in “clumsy thumbsy” as they tried to fill out a nearly microscopic field or navigate to a portion of the page they needed.

According to Adobe, nearly 8 out of 10 consumers would stop engaging with content that doesn’t display well on their device. Even with responsive design, mobile testing can help you understand the nuances of the mobile experience to be certain that your digital properties meet user needs.

Even if your company isn’t mobile-first, it’s important to understand the experience on mobile devices to make sure the experience is consistent throughout the customer journey.

Mobile web testing tips

• Test for clumsy navigation with fingers and thumbs: Google Product Director, Luke Wroblewski suggests a "one thumb, one eyeball" approach to testing on mobile. The idea is that users tend to be distracted when using mobile devices, so can users easily achieve their goals on your mobile site with just one thumb and one eye on the screen? This is a fast and easy way to pinpoint general usability pain points and will also help improve the overall accessibility of your mobile site.
• Observe users interacting with dropdowns: The menu/dropdown experience on mobile websites will be very different than on desktop, so be sure not to miss testing out these details. Menus are critical to helping users find the information they need, so making them easy to use and accessible is important to the overall experience.
• Note how users navigate when scrolling is required: Scrolling is another function that's fairly intuitive on desktop, but not always so with mobile websites. This is where using a combination of screen recording and the test participant's rear-facing camera to capture their facial expressions as they try to navigate using scrolling functions. Don't be afraid to ask participants to scroll more or less than your site will actually require as this will help you benchmark tolerances for scrolling for future updates.

See mobile web testing in action with UserTesting

Remote video URL

Testing real-world experiences

Meeting your customers where they're at is one of the most important factors in creating great experiences. Now, more than ever, people rely on their mobile devices to do much more than talk on the phone or play games. Mobile devices serve as conduits for communication, work, entertainment, healthcare, education, financial services, social services, and much more. 

As societies adapt to a more distanced existence, at least for the foreseeable future, it's more important than ever to understand how people interact with mobile experiences in real-world settings. 

Whether it's an app, mobile web, AR/VR, or any other experience that involves a smartphone, observing those interactions in a natural setting is critical to creating an experience that resonates with and is useful to your users.

Real-world experiences you can test

If you're accustomed to testing in a lab, conducted mostly digital usability studies, or have never tested experiences before, imagining testing in real-world settings can be intimidating—at first—but once you realize that you're simply observing behaviors interacting with devices most of us have within arms reach, 24/7, testing real-world experiences will feel much more natural. Some popular types of real-world experience tests are: 

• Contextual inquiry: participants talk about their process and environment at work or at home
• Personal inventories: participants share pictures of collections of their belongings, e.g., items in their backpacks, purses, suitcases, etc.
• Touchstone tours: participants give you a guided tour of their environment and answer questions about how they organize their space

Real-world testing tips

• Recording guidelines: There are regional guidelines regarding securing consent when recording on location and when in the presence of others. Before launching a study that requires that the participant engage their camera, be sure to research the recording laws for the location and ensure that these guidelines are provided in instructions to the participant. 
• Acknowledge the intimacy of real-world testing: Remember, when conducting real-world testing, you're being invited into a person's private environment. They're essentially welcoming you as a guest into their home or work environment, so be sure to acknowledge that level of intimacy and assure your test participants that you'll treat their feedback with the utmost respect, care, and privacy.

See real-world testing in action with UserTesting

Remote video URL

Mobile testing is part of a customer-centric culture

Mobile is an important part of any successful company's strategy—but it's just one piece of a larger puzzle. At the heart of every successful company is a customer-centric mindset that relies on human insights at every stage of development, on every device, and on every channel to keep creating great experiences for their customers.

Testing your mobile experiences with real users throughout your development cycle will help you discover the drivers of behaviors on mobile, including why users abandon apps, what fuels frequent usage, and how you can improve experiences to drive greater adoption.

The shift in mindset and the understanding that there is a nuance in how students use mobile is an important realization for Udemy. - Michelle Fiesta, UX Researcher, Udemy