How to evaluate a human insight solution

The whole point of fast human insight is to eliminate costly guesswork by delivering insights at the moment of decision-making, so you don’t have to fail fast and you don’t have to revisit decisions later. You should evaluate every step needed to get insights quickly, including:

• How quickly can you create a test? Is a rich set of templates available, and how difficult is it to use them? Can you customize the templates for your company’s needs?
• Are the participants ready to go? Does the system have a robust first-party panel of participants ready to run tests immediately, or will you be forced to work through a third-party panel? If you’re working through a third party, how much will that slow you down?
• Does the vendor offer access to specialized or hard-to-find participants? Can they recruit for you, around the world, with high quality and at speed?
• How quickly do you get results? A good human insight system should deliver answers within a couple of hours on average. Longer lead times will slow down your agile product processes and delay marketing projects.
• How quickly can you analyze test results? There’s no sense running a test quickly if you can’t get to the answers equally fast. Is AI analysis built in? Does it measure both emotion and the content of answers? Will it create video clips for you automatically? How well does it summarize results?

A human insight system is not just about testing software usability. To create a great customer experience you need to be able to quickly understand customer emotions and motivations, and you need to see every aspect of the customer journey, including real-world experiences in addition to online. Here’s what to look for: 

• Can you do both live interviews and self-guided tests? A good human insight system will let you recruit, schedule, record, and transcribe live video interviews. You should be able to schedule those interviews within a day unless you need special recruitment, and during the interview, you should be able to share your screen with the participant and to see their screen and face. The system should also support self-guided tests in which there’s no moderator. 
• Is the interview scheduling and recruitment automatic? Can you recruit and schedule live video interviews automatically through the system, or do you need professional services help? Do you have to pay extra for that?
• Can you test omnichannel experiences? It’s critical that you be able to test every aspect of the customer journey, including complex online-offline interactions like in-store pickup. Participants using smartphones or tablets should be able to record what’s happening on their screens and, in the same test, use the backside camera to record conversations and physical interactions.
• Can you test real-world activities like unboxing and real-world usage, both at home and in other locations? A good human insight solution will let you test almost any real-world experience, including product unboxing and real-world experiences outside the home, whether in a store, work, travel, or other location. The vendor’s professional services team should be able to help you recruit participants for these activities.
• Can you conduct self-interviews? Participants are sometimes more willing to open up on difficult issues when there is not a moderator judging them. A good human insight system makes it easy to have participants interview themselves by recording their faces while they respond to written questions on screen.
• Can you test mobile apps? For many companies, mobile apps are a critical part of the overall value proposition. Can your human insight solution test unreleased mobile apps? Does it handle provisioning of the apps automatically, or do you have to manage that on your own? Can you run the tests in the wild, in locations where a mobile app would typically be used, or does the participant need to be tethered to a computer?
• Can you test marketing material, like creatives and messages? Fast feedback is a great way to ensure that messages and creative elements resonate with customers before you launch them. Your system should be able to test messages and other written content, images, videos, and interactive online experiences. A good human insight system can also test app store listings, including icons, screenshots, and product descriptions.
• Can you test prototypes? How easy is it to test early and late-stage prototypes? Does the solution integrate directly with Figma? Are the tests pre-configured for you? Can you run click tests on early-stage sketches and images?
• Does the system help you improve information architectures? Do you have access to specialized tests for structuring information online, including tree tests and card sorts?
• Can you do surveys? A good human insight system will let you run quantitative surveys alongside your interaction tests.
• Can you get visualizations of complex data? A human insight system can capture large amounts of data on web interactions and emotional reactions. Does the system help you visualize these results, so you can analyze them quickly? A good system should support Sankey diagrams, heat maps, and emotion tagging of transcripts and test results.
• Can you benchmark experiences to track progress and evaluate the competition? Does the benchmarking process incorporate both attitudinal and behavioral data? Are the benchmark calculations performed for you automatically?
• Can you see the participant’s face and hear their voice? You can’t fully diagnose problems and understand reactions unless you can see and hear the participants in addition to viewing their screens.
• Is the system optimized for human insight? Sometimes companies bolt human insight features onto solutions designed for other purposes. Like a Swiss army knife, these systems look powerful but the usefulness of each tool is compromised. Check for limitations built into the system’s design. For example, if you’re forced to bundle a human insight test into a survey, you may not be able to record the entire test, and you’ll miss important insights.

To get the full productivity benefits of fast feedback, human insight can’t be confined to a single working group or department. It’s important that every person creating customer experiences, including people in product and marketing teams, be able to proactively gather feedback on their own work or access historical feedback when they need it. Does your human insight solution support this, and what does it do to prevent poorly structured or misleading tests?

• How usable is the system by non-researchers? How much training do they need, both in terms of understanding the interface and learning how to access or gather feedback? A system designed only for researchers will hinder broad adoption of insights or impose huge training burdens on the company.
• Does the system help you design tests? How many templates are available to make testing easy for marketers, designers, product managers, and others without a research background? Can you customize and add to them? Is there AI assistance for designing screeners?
• Does the system support locked-down templates that resist mistaken edits by non-researchers?
• Can a user be limited to using particular templates or test types? This prevents wasted tests and helps align test types with a user’s job role and expertise.
• Can you create private workgroups whose tests can’t be seen by the rest of the team? This ensures confidentiality when the company is working on sensitive projects.
• Can you create and share pre-built participant groups? Non-researchers often struggle with defining the right audience from whom to collect feedback, so it’s important that they can select from organization-specific, pre-defined groups.
• How many pre-defined demographics are available? A good system should let you automatically select participants by not just age and gender, but also job roles, industries, and other important demographics.
• Are results presented in a way that non-researchers can easily understand? Are there AI analyses and result visualizations that non-researchers can easily understand? Are the emotions highlighted in the results?
• How easily can you collaborate and share insights? What tools are included to help you share the insights you extract from tests? How quickly and easily are you able to extract video clips and create highlight reels? Are time-saving features like transcripts available, and can you generate clips by selecting the precise words you want in the transcripts (as opposed to sentences or paragraphs)? Can you embed clips in common productivity apps?
• Does the solution include an insights hub, a central mechanism to identify and share insights from tests? Does it help you create reports and collaborate on them? Can you import data from third-party app reviews and analyze the data using AI?
• Can you search across the transcripts of all your tests?
• How much support is available, and when is it available? Does the support team have a good reputation for responsiveness? 
• Does the system allow approval flow to ensure test quality prior to launch?
• What do the industry analysts and users say? Is the solution recognized as a leader in human insight? If not, what’s missing and why?

The most beautifully structured test is useless if the participants are uncooperative, unqualified, or fraudulent. You should ensure that your human insight vendor has the size, skills, and infrastructure to protect you from inaccurate and malicious participants.

• Does the vendor have its own, vetted panel of participants? The fraud rate in third party market research panels is between 38% and 70%, according to multiple studies. Does your human insight vendor protect you from these fraudsters? No system is 100% perfect against fraud, but a good human insight system should include a large first-party panel screened rigorously to exclude bots and fraudulent participants, and also tested to ensure that participants understand how to think out loud, so you will get high quality feedback. The panel should also be large enough that testing work can be spread across a large pool of people, so they don’t become habitual testers. You should ask how many tests the average panelist completes before they move on.
• Who does technical support for the participants? Who is responsible for installing recording software on the participants’ devices, and who will do the technical support if the participants have trouble? Who takes responsibility for fixing any problems that arise? Will you get caught between two vendors?
• Is there a “fresh eyes” feature? As an added protection against habitual testers, you should be able to require that no single participant will do more than one study for you in a specified period of time.
• Do you have the opportunity to rate the quality of participants? A good human insight system uses participant ratings to continually improve its first-party panel. Low ratings should trigger automatic replacement of a participant in your test, and should also get a participant removed from the panel.
• Can you ask the participants follow-up questions? A good human insight system will let you contact participants from the first-party panel if you have follow-up questions for them or want to invite them to an additional study.
• Does the vendor have your back when using third-party panels? If you need to sample from a third-party panel, does your human insight vendor take responsibility for vetting the panel? Will they commit to solving any problems you identify, including replacing bad participants? Can they teach you how to detect and exclude fraudsters?
• Is specialized recruiting help available? No panel contains every type of participant you might need. If you have specialized needs, can your human insight vendor help you recruit them? Check if they can recruit overseas, and help you find people with senior titles and specialized job roles.