AWS + UserTesting

Over time, Amazon’s business leaders have made UserTesting a fundamental part of the organization’s product development process. Hearing directly from customers early, and integrating their feedback into the creation of how products look and work, fits snugly with the very first of Amazon's 16 Leadership Principles: "Customer Obsession." 

That principle states: "Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers." This principle sets the tone for the entire company culture.

The leaders of the AWS Customer Experience Organization recognize that you can meet the needs of customers by harmonizing data with real-world sentiment from existing and potential customers. Like Amazon founder Jeff Bezos, Menz believes in the power of the anecdote. “Because the customer is so central to how we work, the key piece is to make sense of the data and to understand what's the most meaningful anecdote, the most meaningful set of results to be able to inject into the discussion. That's where the magic is–to understand that synthesis.”

Sarah Alvarado, UX Research Manager at AWS, explains how the organization’s partnership with UserTesting has helped them work across teams to research, test, develop, and publish some of the world’s most powerful functions for software developers.  

“Over the course of two years, we’ve conducted evaluative research with hundreds of AWS customers and we've also helped to upskill and uplevel the team by giving product manager and designer partners the ability to leverage UserTesting as a platform–to get in front of our customers and ensure that we're gathering feedback early and often to make the experiences better for them.”

AWS has excelled at testing and developing artificial intelligence (AI) functions in particular. 

The AWS Pricing Calculator

The AWS Pricing Calculator is a free web-based tool for existing and prospective customers to create cost estimates for using AWS services. In past years, customers would rely on sales reps to configure their options and show the related costs. Today, customers use the Pricing Calculator to make informed decisions about their investments in AWS by understanding potential costs upfront, comparing different service configurations, and exploring alternative solutions. 

Research showed that customers had significant interest in using the calculator, but when AWS first launched the function the usage levels were surprisingly low. AWS ran experiments with UserTesting to find out what happened. 

According to Laura Faulkner, Principal UX Researcher, “In our user research for the Pricing Calculator, test participants told us they wanted to see how each option affected the price. They wanted to see what we saw and what our specialists saw. With the Pricing Calculator, we were able to add greater transparency of what each option costs and what the impact was on the final price.”

Amazon Kendra 

Amazon Kendra is a powerful search tool that uses AI and machine learning to help organizations more easily build advanced search functions. Kendra indexes databases and documents from repositories like Microsoft SharePoint, Salesforce, ServiceNow, as well as unstructured data like PDFs, Word documents, and emails. Kendra’s algorithms then analyze these documents to understand their context and integrate that information into search results. 

Once they learned about the pain points from developers, AWS designers created mockups in Figma and brought them into UserTesting. They ran tests with a broad range of subjects, from the smallest solo developers to developers on enormous teams who need streamlined methods to collaborate.

Laura Faulkner describes how running experiments helped the organization build Kendra to function in the ways that customers need. “We worked with UserTesting to run a number of studies over a long period of time. That's a lot of content. We wanted to discover two things: how do we do this well for the builders, but then also have great results for the end users of the searches?”

The early results are intriguing. AWS learned that their developer audience and their technology audience wanted a dashboard. The dashboard could help them track the effectiveness of the searches for users, fine tune the search models, and improve them over time. AWS developed the dashboard in Kendra and noted that customers use the platform to automatically fix or remove dead links and also to track user behavior. Other improvements will come in time, as the team will continue working with UserTesting to increase usage and improve the performance of Amazon Kendra.