7 tips for smoother remote moderated usability studies

Today’s article is a guest post from Devin Harold, UX Research Manager at Verizon. Enjoy! Remote moderated usability testing can be a great way of getting valuable insights quickly without potentially more expensive external facilities or a full lab set-up. While convenient, one must plan carefully to prepare for inevitable disruptions, maximize time during sessions, and keep the ecological validity of the study consistent. Here are seven simple tips to mitigate common issues and keep your research sound!

1.    Create backup plans

Just as one should prepare for in-person research sessions, you must arm yourself with backup plans in case things go wrong (and they certainly will at some point). Typical scenarios which I prepare for are: poor internet connection, limited video connectivity, poor participant audio, delayed session start times and, of course, no-shows. To prepare for poor internet connection and shoddy video connectivity, consider screening participants with these criteria to mitigate potential issues. For example, a question regarding their monthly internet download speeds should help screen out any participants with less-than-adequate bandwidth for video conferencing. During your sessions, if you’re having a hard time understanding participants due to a weak audio connection, have them call into the conference using their cell phone. Make sure you’re well equipped with step-by-step instructions for finding the conference’s audio settings and dial-in numbers—this minimizes any wasted time. Delayed start times and no-shows are always part of recruiting for any type of user experience research. However, there are ways you can prepare. Add buffer time within your test plan for tech checks (highly recommended), introductory material, or instructions before you begin each session. I typically allow for 10 minutes of introduction before jumping into the task script. As always, be sure to over-recruit so you won’t be scrambling if some of your participants don’t show up. In my experience, remote moderated sessions are quite easy to join for participants, so no-show rates are typically lower than in-person studies hosted on-site, where participants need to drive to a facility location and block off a larger chunk of their day. Since I run my studies in accordance with discount usability best practices (testing with five participants), I typically recruit six people for my usability sessions. The extra participant gives me a bit more confidence in my data while also providing a comfortable buffer for any no-shows.

2.    Show up early

As with any meeting, you should aim to show up a bit early. Not unlike other user research sessions, the reason you show up early is to alleviate any wasted time from technological issues and prepare necessary materials for the study. Specifically for remote moderated sessions, you should make sure that both you and the participant show up five to ten minutes before the session start time to do a quick tech check, setting up the video conference and sharing any instructions, documents, or web links with them before you begin.

3.    Interrupting is acceptable—for specific reasons

It is not a best practice to interrupt your participants while they’re speaking, sharing thoughts, or completing tasks. This practice may frustrate participants and lead them to feel disinclined to share information with you. That said, in the event of a technical issue or when the participant doesn’t understand the prompt or instructions given, you may intervene. Politely ask the participant to repeat themselves and clearly explain that their audio cut out. If this is a persistent problem, you should ask them to check their internet network or wifi settings to ensure that their connection is solid. Of course, only ask this of them if you’ve already confirmed that yours is not the problem. This is where screening with a strong internet or wifi connection can prove beneficial. If a participant doesn’t understand the prompt or instructions given, intervene and clearly state that you made a mistake in explaining the prompt. Never make the participant feel as though they misunderstood or they will begin to rely too much on you for additional explanations throughout the remainder of the session. Make a note of what they believe they were being asked and attempt to instruct them further. You should revisit your task scenario or instructions after the session and revise them accordingly. In general with remote moderated sessions, it is OK to interrupt participants if they said something interesting or mentioned something which you need clarification on. In person, it’s easier to follow thoughtful moderation techniques for extracting a participant’s thought process than it is within a video conference; you lose out on subtle body language and nonverbal cues. If you need to be absolutely certain about what you heard or observed (and you can’t simply revisit the recording later on) then interrupting participants in this scenario is justified.

4.    “Unplug” during sessions

This rule should be a given. However, I wish to highlight specific examples of “unplugging” so that you can work to mitigate any risk of interruption or influencing participants. Be sure to turn off all of your notifications, both browser-based and computer-based, so that there aren’t alerts, chimes, or other noises that might prove distracting while giving instructions or observing. Because remote moderated sessions can be so quick and easy to set up, it is easy to forget to block your calendar or inform relevant stakeholders you’ll be “out of pocket” during these times. I find this to be an absolute must when conducting remote research. This means, no matter the fire drill, request, or point of escalation by your team or superiors, they know you’re in the room with customers and cannot be bothered under any circumstances. In addition to blocking my calendar and informing others, I often put a sign outside of my conference room door saying “No interruptions—customer interviews in progress” so that passersby know to be quiet, professional, and courteous.

5.    Use the conference chat

Whether it be the frame of mind, scenario, individual task prompts web/download links, or step-by-step instructions, use the conference chat for providing participants with any necessary information. You should use the chat to post tasks so that you’re teaching participants to rely more on the task and less on you, the moderator, for useful information during the sessions. In my experience, it has proved very beneficial to provide participants with individual tasks using the chat. Much like tasks given in unmoderated sessions, this allows them to revisit each activity/prompt so that they may work at their own pace and revisit them if they simply forget (memory is fallible, you know!). This is especially important if you require participants to use a faux credit card or login credentials while completing tasks.

6.    It’s OK to take notes

Taking notes while moderating interviews or usability sessions have long been a controversial topic, with the general consensus being that it is bad practice. This conclusion is usually found due to an invisible barrier which is introduced between you and the participant while you’re furiously scribbling in a notebook or typing on a computer. Remote moderated studies don’t impose such barriers; you’re already conversing via computer screen many miles away. Be sure that when you’re taking notes, you’re paying full attention to the participant at all times and only documenting key interactions or quotes. While taking these high-level notes is OK, it is best practice to invite a colleague to observe sessions for more detailed notes such as time logs or the participant’s stream of consciousness.

7.    Don’t do Gen Pop recruitment

Even though remote moderated sessions can be so easily conducted within tight timelines, don’t fall into the trap of recruiting only the general population. You still need to make sure you’re recruiting representative users; these are the people who will have the true motive and mindset for completing real tasks on the system being tested. As we know, people pay attention to different things depending on their goal—if the goal of the participant is simply to do your task without motive, they’re likely to pay attention to very different things than your real users. Representative users often pay closer attention to content, connotational meanings, and associations when they’re navigating and completing tasks. Given this, it is absolutely imperative that you continue to recruit representative users, even if you’re conducting “quick-tests” to evaluate only very general flows or the big “roadblocks” within an experience. Follow these seven tips for conducting remotely-moderated user research, and you will find that your sessions not only will have richer insights but will go much more smoothly for yourself, your stakeholders and, ultimately, your participants as well.

Want to learn more?

To learn how UserTesting can help you understand your customers through on-demand human insights, contact us here.