Generative vs. evaluation research

​

There are many different ways to conduct user experience (UX) research. Qualitative vs. quantitative, attitudinal vs. behavioral, and remote vs. in-person are just a few research categories. 

In this post, we're examining generative vs. evaluative research. 

In a nutshell:

• Generative research: Typically uses qualitative methods to understand user needs and motivations to inspire design and product development innovation.
• Evaluative research: Typically uses quantitative methods to assess the success or failure of a specific product/feature and identify necessary improvements.

Below, we examine each type in depth, explaining why it's important and some of its methodologies. We also delve into the differences between generative and evaluative research methods, examining the pros and cons of each and discussing how to justify UX research. 

Generative research

What is generative research?

Generative research, or discovery research, is a methodology researchers commonly employ in design and product development. Its primary focus is understanding user behavior, needs, and motivations to generate insights and ideas for new products.

Generative research typically employs qualitative methods, such as interviews, observations, and ethnographic studies. The goal is to uncover deep insights into users' experiences, preferences, and pain points, which user experience teams can use to inspire and guide the design and development process.

Generative research aims to look to the world around you to find opportunities for solutions and innovation. These solutions could be new products or experiences or updates or improvements to existing ones.

To identify new and innovative solutions, you must define the problem you are trying to solve. This requires you to understand how people live, including their environments, behaviors, attitudes/opinions, and perceptions.

When conducting generative research, the most important thing is to keep an open mind—you may not know what problem you’re trying to solve yet.

Why is generative research important?

"The best research you can do is talk to people." 

— Terry Pratchett, author, humorist, and satirist

Generative research is important for several reasons:

• Understanding user needs: Generative research provides a deeper understanding of users' behaviors, preferences, and needs. Uncovering insights about users' motivations and goals allows you to develop products that better meet their needs.
• Inspiring innovation: Generative research often involves exploring new and diverse perspectives, which can inspire innovation and creativity. By collecting voice of customer (VoC) data and gaining insights into users' experiences and challenges, teams can create innovative solutions that address unmet needs or solve existing problems in novel ways.
• Informing design decisions: The insights you gather from generative research can inform design decisions by better understanding users' needs and wants, resulting in more user-centered designs.
• Reducing risk: By conducting generative research early in the design and development process, you can reduce the risk of developing a product that fails to resonate with your target audience. Understanding users' preferences and behaviors early on helps you make more informed decisions and reduce potential risks.
• Validating assumptions: When you conduct generative research, you can validate assumptions and hypotheses about user behavior and preferences. Gathering real-life data and insights can verify whether your assumptions are accurate so your team can adjust your strategies accordingly.
• Creating empathy: Engaging in generative research fosters empathy toward your users. It allows you to immerse yourself in their experiences and understand their perspectives more deeply. This empathy is essential for creating products that effectively address users' needs.

When should you conduct generative research?

Product development and UX teams often conduct generative research during the early stages of a project. This is when exploring and understanding the problem space is needed before moving on to more specific and targeted research activities. Through generative research, these teams gain a deeper understanding of their target audience and the context in which their users operate, ultimately leading to more effective and user-centric solutions.

Generative research methodologies

Generative research encompasses a variety of techniques for exploring user behavior, preferences, and needs. These and other methods can help you better understand user perspectives and generate valuable insights for developing solutions that satisfy users.

User interviews

An in-depth interview or live conversation involves one-on-one discussions with users to understand their experiences, needs, and preferences regarding your product. These interviews delve into specific topics, allowing you to gather in-depth qualitative insights and uncover user motivations and pain points.

Field studies

During field studies, researchers observe and interact with users in their natural environment to gain firsthand insights into their behaviors, contexts, and product interactions. By immersing yourself in the user's environment, you can understand their real-world experiences, challenges, and needs, gathering rich qualitative data to inform design decisions.

Ethnographic studies

An ethnographic study involves immersion and observation within a specific cultural or social context to understand a community or group's behaviors, practices, and values. Researchers adopt a participant-observer role to gain insight into users' daily lives, routines, and cultural influences, providing a nuanced understanding for designing culturally relevant products or services.

Open card sorting

The open card sorting method contextualizes how users categorize and organize information or content. Researchers give participants cards representing content items and ask them to group the cards meaningfully. The users must also name each grouping. This helps identify users' mental models and preferences for organizing information, informing the structure and navigation of digital products like apps or websites.

Open-ended survey questions

With open-ended survey questions, test participants provide detailed, free-text responses to questions rather than selecting from a predefined list of options. This methodology enables researchers to gather qualitative data and insights into users' thoughts, opinions, and experiences in their own words. With open-ended survey questions, you receive rich, contextual information that complements quantitative survey data, offering a deeper understanding of user perspectives.

Evaluation research

What is evaluation research?

Evaluation research, also referred to as evaluative research, can be defined as a research method used for assessing a specific problem to ensure usability and ground it in the wants, needs, and desires of real people.

Also known as evaluation or summative research, evaluative research aims to assess the effectiveness, efficiency, or impact of a product, program, policy, or process. Unlike generative research, which focuses on understanding user needs and generating insights, evaluative research centers around evaluating and measuring the outcomes or results of a particular initiative.

This type of research often involves using quantitative methods, such as closed-ended survey questions or heuristic reviews, although researchers may also use qualitative methods in some cases. The goal of evaluative research is to provide evidence-based insights into the success or failure of a specific product or feature and identify improvement areas.

Evaluative research allows design, development, and UX teams to provide stakeholders with evidence-based information about the effectiveness and impact of their projects, helping them make informed decisions and improve their product's quality.

Why is evaluative research important?

"I believe in innovation and that the way you get innovation is you fund research and you learn the basic facts."

— Bill Gates, entrepreneur, philanthropist, and founder of Microsoft

Evaluative research should always be a part of the iterative design process. Getting designs in the hands of our users as soon—and as often—as possible ensures that the experience will be shaped and refined to truly meet customer needs and expectations.

There are many reasons evaluative research is valuable, including:

1. Assessing effectiveness: Evaluative research helps determine the effectiveness of a product in achieving its intended objectives. It provides evidence-based insights into whether the product produces the desired results.
2. Measuring impact: This research method allows researchers to measure the effects of their product on its target audience or stakeholders. 
3. Identifying areas to improve: Evaluative research highlights a product's strengths and weaknesses, identifying areas for improvement and providing valuable feedback you can use to make adjustments, refinements, or enhancements.
4. Making data-informed decisions: Findings from evaluative research inform decision-making by providing stakeholders with evidence-based information about the product's success or failure. This helps your team to make informed decisions about whether to continue, modify, or terminate the feature or product.
5. Ensuring accountability: Evaluative research promotes accountability by providing stakeholders with data and evidence to evaluate an initiative's performance and impact. It helps hold individuals or organizations accountable for the resources invested and the outcomes achieved.
6. Supporting iterative design: Evaluative research promotes continuous improvement by providing feedback loops that enable iterative refinement and optimization. It facilitates learning from successes and failures to enhance the current project and future initiatives.

When should you conduct evaluation research?

Typically, researchers conduct evaluative research during the later stages of the product development cycle after developing a prototype or product. This research helps identify usability issues by gathering user feedback and measuring the product's success rate in meeting its intended goals or objectives. The team then uses this data to make product improvements or refinements before its launch.

Evaluative research methodologies

Within the scope of evaluative research, multiple techniques aim to assess a product's effectiveness, impact, and user satisfaction. These methodologies use various quantitative and qualitative methods to measure outcomes, identify areas for improvement, and inform decision-making. Some examples of evaluative research methodologies include:

Closed-ended survey questions

How you conduct a survey and your questions can be categorized as either generative or evaluative. Utilizing predefined answer options, closed-ended survey questions streamline responses for quantitative analysis, facilitating efficient data collection on specific topics or variables and enabling cross-comparisons among participants.

Tree testing

Employed in usability testing, tree testing evaluates the navigational structure of websites or digital interfaces by presenting participants with simplified representations of site hierarchies. This enables users to locate content or complete tasks, thereby identifying usability issues for optimization.

Closed card sorting

In contrast to open card sorting, closed card sorting uses predefined categories to assess the effectiveness of existing hierarchical structures for organizing content. This can inform decisions about content organization and labeling.

A/B testing

A/B testing, or split testing, compares product variations to determine which performs best. This practice provides insights into effective design, content, or feature variations based on objective metrics such as conversion rates or user engagement.

Heuristic review

Conducted by expert evaluators against established usability principles or heuristics, heuristic reviews identify potential usability issues, offering insight into usability problems and areas to improve based on recognized usability principles.

What are the differences between generative and evaluative research?

"Generative research dives into 'What problem might we solve?' while evaluative research asks, 'How well is our solution working?'"

— Erika Hall, Co-Founder, Mule Design Studio

To summarize, generative research:

• Focuses on understanding user needs, behaviors, and motivations
• It is typically conducted at the early stages of a project to inspire innovation and inform the design process
• Uses qualitative methods such as user interviews, field studies, and ethnographic studies
• Generates insights and ideas for developing user-centered solutions

Alternatively, evaluative research:

• Centers around assessing a product's effectiveness, impact, and user satisfaction
• It is usually conducted after implementation to inform decision-making and guide improvements
• Uses quantitative and qualitative methods such as tree testing, A/B testing, and heuristic reviews
• Evaluates outcomes and the effectiveness of features and products

How to justify UX research

"In an increasingly technological world, designing products with real people in mind helps us make sure that technology integrates in our lives in a human way."

— Leah Buley, Senior Director of Consumer Insights and User Research at Lovevery and author of The User Experience Team of One: A Research and Design Survival Guide

Justifying UX research—which can involve both generative and evaluation research—means demonstrating its value in improving user satisfaction, enhancing product usability, and ultimately contributing to the product's success. 

UX research can benefit your organization in several ways, including:

1. User-centered design: UX research allows you to design products with users' needs, preferences, and behaviors in mind.
2. Reduced risk: By uncovering insights throughout the design process, you can identify usability issues and design flaws, mitigating the risk of developing a product that fails to resonate with users.
3. Improved conversion and retention: Optimizing UX can improve conversion and retention rates by enhancing the user journey and increasing user loyalty and engagement.
4. Enhanced brand reputation: A positive UX contributes to a positive brand reputation, differentiating you from your competitors.
5. Increased return on investment (ROI): Investing in UX research can lead to a higher ROI by improving your product's overall performance and success.
6. Stakeholder alignment: UX research facilitates informed decision-making by providing evidence-based insights into user preferences and improvement.

By highlighting these benefits, you can make a compelling case for investing in UX research as an integral part of the product development process.

Generative vs. evaluative research: Why not both?

The choice between generative and evaluative research methods hinges on the specific goals and stage of the project. While generative research delves into understanding user needs and motivations to inspire innovation, evaluative research focuses on assessing a product's effectiveness and impact. 

Both approaches are valuable in informing design decisions and improving user satisfaction. UserTesting participant recruitment can help with both types of research, providing diverse perspectives and meaningful insights.

Ultimately, by marrying generative and evaluative research methodologies, UX teams can create user-centered solutions that meet user needs and organizational objectives, leading to successful product outcomes.

This post was updated 4/9/2024.

​