The things people tell you when you’re not there

Recently, our friends at NerdWallet published a fantastic post on Medium discussing some impactful discoveries that their team made while using the UserTesting platform to conduct remote interviews. One key insight they gained was the advantage of removing the interviewer from the discussion. This was especially beneficial when it came to discussing sensitive information, like financial decisions—something that’s important to the folks at NerdWallet. As a result, the NerdWallet team gained a new perspective on how to empathize with their customers and still achieve measurable, actionable results. We think that any way an organization can incorporate empathy into the customer’s journey is a worthy endeavor and wanted to share this great advice on gathering human insights from the NerdWallet team. What follows is NerdWallet’s article (shared with permission). Enjoy!

The things people tell you when you’re not there

UserTesting isn’t a tool typically recommended for interviewing, but with some tweaks, it can throw open the doors on discovery research

As researchers, we’ve spent hundreds of hours interviewing people 1-1. Getting to spend time hearing people speak honestly about their lives is one of the best parts of our jobs. But comprehensive interview projects take a lot of time — and we don’t always have it, working on tight timelines in a fast-moving company. So what do we do if we need to quickly understand how people talk about an issue we’re newly tackling, or if we need to hear from people outside our home base city (and have 48 hours and no travel budget)? At NerdWallet, we’ve started using UserTesting— the remote, unmoderated platform typically used to get quick feedback from a pool of paid testers — to conduct user interviews. As researchers who built careers on talking to people face-to-face, we were skeptical: “But … You can’t follow up! And you have to know all your questions in advance!” But once we set those doubts aside and tried it, we found that unmoderated interviewing can elicit deep, nuanced, and natural responses from participants. It turns out there’s an entirely different, special kind of intimacy that emerges when someone tells stories alone to their computer. It doesn’t replace in-person research, but it can be an informative, low-friction supplement and should be a tool for every fast-moving team to consider. Here’s how we do it.

When and Why to Consider this Method

When you want interview data ASAP

Unmoderated interviews let you time travel, getting answers back in literally fractions of the time an in-person project would take. Specifically, with this method you can:

• Recruit easy-to-find groups easily and quickly. Unmoderated tools often have their own panels (and some also let you use your own recruits!). On UserTesting, once you launch a screener to their panel using their survey tool, there’s no time between people filling out the screener and doing the interview — so there’s no need to call or send reminders, either.
• Pilot quickly. It’s simple to launch an interview to just one or two people while you’re still working through your script. This can be a great way to see which questions land, and which ones people answer with a surface-level response — or say something that shows they’re not understanding what you’re getting at.
• Run multiple interviews at the same time. It’s basically a cloning machine: If you put out a 15-minute script on UserTesting to 15 participants, all 15 can theoretically answer your questions during the same 15 minutes. <insert mind blown gif here>

When you want honest directional data

• Minimize the observer effect. This was a big a-ha moment for us. At NerdWallet, we talk to people about money. Even with the best research interviewing techniques, there are moments when talking about people’s financial decisions, fears, and missteps gets awkward. They have to look you in the eye and tell you something very personal. But when they’re at home, alone, talking to a screen, nobody is staring back. “Wow, I just learned something about myself” is something we’ve heard more than once.
• Get a baseline understanding in new areas. When you’re trying to understand a new domain, almost everything people tell you is helpful. This is a great way to get a foundational understanding of a group or topic area before investing further. You’ll go into future research with a stronger foundation, ready to ask probing questions.

When you want to focus on efficient but thorough reporting

• Organize and find patterns easily. In semi-structured interviewing, notes become harder to organize, especially if you’re simultaneously the interviewer and notetaker. With these more-structured scripts, we can take notes easily in a spreadsheet, dedicating a line to the questions and adding in responses from every participant. It makes seeing themes across participants much easier.
• Listen back faster. The downside of launching multiple interviews at the same time: There’s (probably) still just one of you to listen to them. But with many unmoderated tools, you can speed up playback with a high level of control (0.1x at a time, on UserTesting). It’s great for getting through less-data-rich parts of interviews, and you can still slow down for the details.
• Easily pull clips to show your team. On UserTesting in particular, the clip-maker is great for grabbing and organizing short snippets of video or audio you want to share with your team. It can make for an amazing podcast-like listening experience; one of our team members told us she likes to play them for herself while cooking dinner.

How to do it well

OK, you’re sold. It’s the perfect time for us to confess to a few limitations. By definition, it’s not an open conversation — you’re not there to ask “why?” or follow up on confusing answers, and you can’t pick up on physical cues. So we’ve created some internal best practices to get the most out of the method. (Ours are specific to UserTesting but should be adaptable to any other unmoderated platform you use.)

Give great directions

• Work your intro. The instructions participants read at the start of studies are the best way to set expectations and control the interview flow without being in the room. We like to remind people that there are no right or wrong answers, and that it’s fine if they don’t know or remember every detail about something that we ask — their stories will still be valuable! We’ll also explain anything important about the overall mindset or session flow — e.g. “The first 5 questions will focus on your financial history, and the final 5 will focus on actions you’re taking right now.”
• Structure smartly. Unlike a moderated interview, where the conversation has the potential to take many directions, this one is bound by the structure you give it. We like to start the session with an open-ended question and follow it up with directed — not leading, but specific — questions that guide the participant to dig deeper in certain areas. It’s OK if participants discuss the same things twice, but we also explicitly grant permission to skip questions if the answers would be redundant.
• Guide the time. We’ve found that some participants err on the side of being too thorough in early open-ended questions, and then we run out of time. (To be fair, this is a problem in moderated interviews too!) In addition to giving participants control to move on from questions they answered earlier, we’ll add cues to remind participants that a long answer isn’t always better. For example, if your question is about what part of vehicle ownership causes them the most stress, you might add “Don’t worry about listing everything, just focus on the top few!” In a later question, you can add time for people to come back to anything they missed. If you’re using a platform with firm session lengths, you might consider setting general time guidelines — for example, “please try to spend no more than 5 minutes on the first 5 questions.”
• Pilot your script. How do you get those time guidelines? The same way you get them for an in-person interview: pilot, pilot, pilot. Run through your study — in person, or to a single user or small set on your unmoderated platform of choice. Think through all the different ways people might interpret your wording and rewrite when necessary. And don’t be afraid to clarify or write follow-ups: Use the instructions to define boundaries or possibilities if you’re worried about participants not understanding your intent.

So Many Stories, So Fast

Unmoderated interviews can be great for moving quickly to get insights. You won’t miss out on the personal stories, either. In spite of us being nothing more than a screen to talk to, we’ve heard firsthand stories from people struggling to save in the face of student debt; of a business owner going from wealth to just making minimum payments; of home buyers dreaming of a home, but blocked by their credit score and financial decisions long past. These stories can still help your team build empathy and understand diverse perspectives, without you or your interviewees having to share the same space or time. Caitlin Barta and Kimra McPherson are user researchers at NerdWallet, where we help people get the most out of their money.

Want to learn more?

Find out more about NerdWallet and how they help tailor advice, content, and tools to ensure people are getting more from their money, here. If you’d like to learn more about how UserTesting can help you understand your customers through on-demand human insights, contact us at support@usertesting.com.