How PayPal increased credibility and confidence with multiple research methods

With 113.2 million active accounts, every UX-related decision has a significant impact. Deborah and her team, like many other UX researchers, face two main challenges:

1. Delivering quality, credible results to stakeholders and decision-makers;
2. Getting it all done quickly and efficiently.

Using data to tell a complete story

PayPal’s research team uses multiple research methods to provide a complete story about the customer experience and to gain credibility with their decision-makers.

“Sometimes I go into a room and present to stakeholders the usability study results with 10 participants. However, those executives want to know what is beyond those 10 participants. So triangulation, whether through remote usability testing or weblogs, allows me to confidently say that this problem holds true in the entire world with a sample of 200, 400, or even 1000 users,” says Deborah.

However, where do you get the time, money, and resources to do all this research?

Deborah says that she gets requests for research that needs to be done quickly and with limited resources. On top of that, PayPal has a small team of 6 UX researchers who are expected to deliver 42 research studies, talk to 413 participants, and field 271,244 touchpoint surveys—all within six months.

Problem: Reducing the number of support calls

PayPal needed to reduce the number of customer support calls from people who wanted to find out how to service their accounts better. Most calls involved users wanting to update their account information, such as email, password, or credit card information. The goal for the research team was to make it easier for these folks to self-serve themselves online.

Initially, this was the page where these tasks could be completed. It looked more like a link farm, according to Deborah.

PayPal’s UX Research team needed to improve the findability of this page and to ensure that its users could use this page to complete the intended tasks.

Creating a research plan

As a first step, PayPal’s research team wanted to make sure that they truly identified the top tasks that users came to perform upon hitting the profile page. Secondly, they needed to understand the users’ mental model for servicing their accounts.

And lastly, upon using the inputs from the steps above to create an alternative design, they wanted to gather qualitative feedback on the new design.

Tackling the methodology part

PayPal uses a Tracker Survey (similar to True Intent Studies) to identify the top tasks users are engaging in and whether they are successful. They also look at web traffic and weblogs to confirm these top tasks (SiteCatalyst data).

For mental models and organizing concepts, they use card sorts to see how people group items. Finally, to gather qualitative feedback, they run a usability test.

Executing the research

Since Paypal had a tracker survey + web traffic data, additional time and cost were not needed for identifying the top tasks. Qualitative feedback is very important to Paypal. As such, they didn’t mind spending money on a traditional in-lab usability test.

However, when it came to card sorting, they had to figure out how to be more efficient with their resources. As such, they wanted to study how a remote unmoderated card sort compared with a traditional in-lab one.

“With remote unmoderated card sort, we can have more participants and spend less time traveling, setting up the lab, and recruiting users. We also save time on the analysis of the gathered data. Online card sorting tools do a great job of outputting a nice tree diagram or some other type of quantitative measures to help make the decision quicker. In the lab, you may have to do some manual work. Also, in-lab testing can get rather expensive, whereas remote unmoderated tools can save a lot of money,” says Deborah.

The UX team thought it was important to get quantitative feedback since they would use the card sort input into the design that would be tested with actual users in step 3, shown above. More feedback and larger sample sizes would give them more confidence to take the findings back to the design team to iterate and design on.

Ultimately, they chose to use UserZoom’s remote unmoderated card sorting solution for this project. As such, they were able to deliver a research study that was cost and time-efficient. They were able to get the design that they needed and to push it out of the door on time.

Deborah left everyone with the following parting words at the end of the webinar: “Here is the 10% that you should go away remembering—triangulation. Triangulation and using multiple research methods tell a more comprehensive story. It increases the credibility of the research and the decision-makers confidence. And secondly, be sure to utilize tools to execute efficiently. There are several research tools out there, and you should consider using them appropriately to save time, money, and heartache.”