How the UserTesting Contributor Network ensures high-quality feedback

Delivering world-class customer experiences requires you to know your customers and their needs so that you can meet or, even better, exceed their expectations. At UserTesting, our Human Insight Platform enables organizations to do exactly this. By gathering feedback from the UserTesting Contributor Network organizations are able to build products, services, and experiences that their customers love.

The UserTesting Contributor Network is composed of real people from around the world and is large enough to help you find your exact target audience. We carefully vet every contributor in our network to ensure you get high-quality feedback, every time. So whether you’re building the next best app, testing your marketing messaging, improving and iterating on your digital experiences, or doing something completely different, we make it our mission to provide you with the insights you need to build the next great customer experience.

85%+ rated sessions receive 4 or 5 stars from customers.

How bots and cheaters influence data quality

The demand for online research has grown globally in recent years. This growth has led to a recently reported increase in fraudulent activity related to bots and cheaters. Bots are programs that are built to answer surveys programmatically. Whereas cheaters are participants on other panels who may not respond truthfully and actively try to “game” the system to get more testing opportunities. 

Both bots and cheaters can affect data quality dramatically by undermining response quality, generally, while also impacting associated quality metrics like success rates, task times, overall ratings, etc. 

How the UserTesting Contributor Network ensures unparalleled quality 

We go to great lengths to ensure that the results provided by our contributors are of the highest quality. With a rigorous focus on integrity and relevance, we provide an industry-best customer feedback system with contributors who meet your expectations. 

Here is the process we follow to ensure our network is composed of the best contributors:

1. Rigorous application screening

While the application process for becoming a contributor may be simple, our guidelines to manage quality and acceptance criteria are stringent. To apply, prospective contributors complete a three-step process. They have to enter and verify their email, complete a profile, and submit a practice test. Our team then reviews every practice test to ensure that each applicant meets our high-quality standards.

Our reviewers ensure that each applicant has the technical abilities to become a contributor. In terms of qualitative assessments, our reviewers check an applicant’s ability to follow directions set by our practice test, as this is a critical ability for providing high-quality feedback during real tests. Finally, our reviewers ensure that the applicant is articulate and skilled at speaking their thoughts aloud as they perform any tasks required of them. While this may seem easy, after reviewing hundreds of thousands of applications each year, it’s more difficult to do than one may think.

If an applicant passes all of these checks, they’re welcomed to the Contributor Network where we conduct one additional quality check. 

2. First test review and scoring 

The applicant screening process helps us retain only the best applicants into our network of contributors. However, the first test review and scoring process allow us to nurture them and provide personalized feedback to ensure they produce the highest quality feedback to customers. To do this, our Operations team manually reviews the first test of every new contributor and provides individual feedback to them!

This step also acts as a final quality check and helps us root out anyone who may have passed the first screening but misrepresented themselves or otherwise not complied with our terms of use—though this is rare. 

3. Customer rating score

An additional tool to help us curate the network and consistently manage quality is the customer rating score. Customers can rate the quality of any video they receive on a 1-5 scale, with 5 being the highest quality. We encourage our customers to rate contributors as often as possible. Doing so allows our customers to participate in the process of ensuring that contributor quality remains high.

Customer Rating Scores are combined with scores from our Operations team to create a composite rating which is used by our algorithms to match tests with contributors. This allows us to ensure our customers receive the best possible feedback from highly ranked contributors. 

4. Ongoing fraud prevention and quality review 

In addition to the measures above, our Operations team has several ongoing measures to help prevent fraud and identify contributors who display any quality issues.

To prevent fraud, we employ multiple commercial technology tools to spot fraudulent activity or duplicate contributor accounts. The Operations team also upholds the Contributor Terms and Conditions and will immediately remove contributors who are found to have violated these Terms or have demonstrated repeated quality issues. 

From a quality perspective, any session that a customer rates below 3 out of 5 stars is automatically flagged for manual review by our Operations team. Contributors may receive additional guidance or face removal from the network at the discretion of our team.

High-quality contributors means better insights for customers

In order to put our customers first, it’s critical that we provide them with the best possible experience gathering feedback. And doing this requires us to ensure that our network is meeting (and exceeding) our customers’ expectations. 

“UserTesting’s panel quality is always fantastic! It’s significantly better than competitors’.” -Benjamin Hoggan, Director of De-risking, University of Notre Dame

Benjamin continued: “Before we switched to UserTesting, I historically kept to in-person interviews because of the extreme specificity in criteria that we need for our interview targets. One of my very first tests was to see if I could find just one very hard-to-find participant with specific healthcare technology experience: a lab tech who performed exosome isolation and purification. Imagine my surprise when I found not one, but THREE in UserTesting’s network! Hard to beat that. 400+ tests later, we couldn’t be happier with UserTesting.”

UserTesting is committed to providing the highest quality network available—we’re continuously improving and innovating upon the processes you see here. We do all of this to ensure brands can connect with their target audiences—who can then provide the highest quality feedback and insights.