What happens when traditional UX methods can’t keep up with cutting-edge AI-driven products? According to Dr. Sam Howard, Head of UX at Curio, it’s time to innovate. On a recent episode of the Insights Unlocked podcast, Sam shared how his team is rethinking UX research to design products that truly resonate with users in today’s fast-evolving digital landscape.

With themes ranging from user-centered design to the impact of AI on research methodologies, Sam’s conversation with UserTesting’s Blair Fraser offers valuable takeaways for UX professionals, product leaders, and anyone shaping customer experiences in the AI age.

The "Many Wizards of Oz" method: a game-changer for UX

Traditional UX tools like wireframes and prototypes are staples of the design process, but they have limitations when applied to AI-driven products. “When you’re designing with AI, the interface becomes just the container for the experience. Static prototypes can’t replicate the dynamic, real-time interactions powered by AI,” Sam said. 

To address this, Sam suggests what he calls the "Many Wizards of Oz" method, which builds on the classic Wizard of Oz research technique. While the original method involves researchers simulating system responses, Sam’s approach uses live AI outputs to give users an authentic feel for the product experience.

Remote video URL

For example, imagine testing an AI-powered cooking app that tailors meal plans based on dietary needs, schedules, and preferences. Instead of presenting users with static mockups, the Many Wizards of Oz method would collect real input, feed it into AI, and return dynamic outputs for users to explore.

This approach allows researchers to test not only the interface but also the AI’s ability to deliver value.

What is the Wizard of Oz methodology?

The Wizard of Oz methodology is a research technique where participants interact with a system that seems fully functional but is secretly operated by a researcher or “wizard” behind the scenes. This allows teams to simulate and test experiences before developing the actual technology, gathering valuable user feedback early in the process. As Sam explains, “The Wizard of Oz method helps you replicate what the system would do in reality, giving users a real sense of the experience before it’s fully built.”

What are the benefits of using Wizard of Oz testing?

Using the Wizard of Oz testing methodology offers several key benefits, particularly when developing complex products:

1. Early Validation of Concepts: By simulating functionality, teams can test ideas and user interactions before investing in full development. This saves time and resources by ensuring the concept aligns with user needs early in the process.
2. Realistic User Feedback: As Sam noted, “The Wizard of Oz method helps you replicate what the system would do in reality, giving users a real sense of the experience before it’s fully built.” This approach ensures feedback reflects the intended user experience, even if the backend isn't finalized.
3. Rapid Iteration: Researchers can quickly tweak the simulated experience based on user reactions, allowing for iterative testing without needing to rebuild or redevelop.
4. Uncovering Edge Cases: By observing how users interact with a simulated system, teams can identify unexpected behaviors or needs that might not emerge in static or linear prototypes.
5. Stakeholder Engagement: The methodology produces tangible user insights and realistic interactions that are easier to showcase to stakeholders, helping build buy-in and empathy for user needs.

When should you employ the Wizard of Oz methodology

The Wizard of Oz methodology is particularly effective in several scenarios, especially when working on innovative products:

1. Early-Stage Development: Use this method when you have a concept but lack the fully developed technology or backend systems. As Sam noted, it’s ideal for “replicating what the system would do in reality, giving users a real sense of the experience before it’s fully built.”
2. Complex Systems with Multiple Variables: If your product has many potential user paths or outputs, the methodology allows you to test scenarios and user reactions before coding every possible interaction.
3. Innovative or Unfamiliar Features: When introducing new, unfamiliar functionality, this approach helps test user responses and uncover insights that might not surface in standard usability tests.
4. When Quick Iteration Is Needed: The method allows rapid changes based on user feedback since the researcher controls the simulated backend, making it easier to iterate without rebuilding prototypes.

Employing the Wizard of Oz methodology at the right time enables teams to validate ideas, refine designs, and gain valuable user insights before committing to development, ensuring products are both innovative and user-centered.

When would you employ the "Many Wizards of Oz" methodology

The "Many Wizards of Oz" methodology, as described by Dr. Sam Howard, is particularly suited for testing experiences where AI plays a central role and user interactions are highly dynamic or non-linear. Key scenarios for employing this method include:

1. AI-Driven Products with Complex Outputs: When a product relies on AI to generate personalized or variable responses, such as a news curation tool or a meal-planning app as discussed in the interview, this approach allows teams to simulate those outputs in real time. “Testing prototypes with tools like Figma becomes a lot less useful because you can’t replicate what the AI is doing,” Sam explained.
2. Early-Stage Testing for Generative AI Features: If you’re working on a product that requires tailored responses based on user data, this methodology lets you experiment with prompting structures and refine the AI's outputs to match user expectations before building a full system.
3. Validating Personalized Experiences: For products that aim to provide deeply personalized user journeys, the Many Wizards of Oz method gathers real inputs from users, processes them with existing AI models, and delivers realistic outputs, ensuring a more authentic test environment.
4. When User Feedback on the AI’s Functionality Is Crucial: As Sam shared, “The interface becomes just the container for the information,” meaning the real value lies in testing the AI-driven core of the experience.
5. Cross-Functional Collaboration: This methodology requires close collaboration between UX researchers, designers, and backend or machine learning engineers, making it suitable for teams ready to work in iterative, cross-functional settings.

The Many Wizards of Oz method shines when you need to explore how AI-driven systems interact with users in real-world scenarios, offering actionable insights that static prototypes or isolated AI testing can't provide.

Tackling the challenges of AI-driven UX

While AI offers incredible potential, it also presents unique challenges for UX professionals. One of the biggest? Bridging the gap between what AI can do and what users need.

Sam shared his perspective on how to manage this balance, “When exciting technologies come around, there’s a temptation to focus on what’s possible instead of what’s needed. That’s why embedding user insights from the outset is critical.”

Remote video URL

Curio’s approach involves targeting early adopters—those most ready to embrace new technologies—and designing solutions around their specific pain points. For example, research shows that 4 in 10 people actively avoid news due to feelings of anxiety and being overwhelmed. Curio aims to address these issues by combining AI personalization with trusted sources, ensuring content is both relevant and reliable.

This focus on early adopters mirrors the advice found on UserTesting’s blog: understanding behavioral archetypes, rather than rigid personas, can open up new ways to approach user segmentation.

Bringing stakeholders along the journey

Convincing stakeholders of the value of user research is a common hurdle for UX teams, and it’s even more pronounced when working with complex AI-driven projects. Sam highlighted the need to align qualitative insights with quantitative data, “It’s not just about presenting answers—it’s about showing where the numbers don’t explain everything and how qualitative insights fill those gaps.”

Another effective strategy? Using visual storytelling. Sam advocates creating short, impactful videos of user sessions, complete with voiceovers and context, to help stakeholders empathize with users. 

“Video is a powerful tool for bridging the empathy gap. It’s far more engaging than a slide deck and gives stakeholders a firsthand look at what users experience,” he said. 

The future of UX Research in an AI world

As AI continues to transform industries, UX teams must rethink their methodologies to keep pace. From the Many Wizards of Oz method to grounding decisions in user insights, Sam’s approach offers a blueprint for success.

“AI is going to change how we work—and that’s exciting. But it’s also a chance to rethink how we bring real value to users,” said Sam.

Remote video URL

Looking ahead, Sam believes AI will fundamentally reshape how UX professionals approach research. From evolving methods to closer collaboration with engineers, adapting to AI’s demands is essential.

He also warns of the potential pitfalls of focusing too much on AI as a feature. Instead, he urges teams to prioritize outcomes, “AI is the method, not the selling point. The user cares about the result—how the product makes their life better—not the technology behind it.”

Related Resources:

• Sam on LinkedIn
• Blair on LinkedIn
• Curio’s website: Learn more about the AI-driven personal news companion
• Sam’s Substack: The Singularity Complex.
• UserTesting's Vision for Artificial Intelligence: This blog post explores UserTesting's approach to embedding AI and machine learning into experience research, aligning with the interview's discussion on AI's impact on UX methodologies.
• How to Test AI Experiences (Part 1): This article offers practical guidance on evaluating AI-driven user experiences, complementing Sam's "Many Wizards of Oz" method for simulating AI interactions during research.
• Video Highlight Reels: 5 Best Practices for Sharing Insight: This blog post discusses effective ways to communicate user feedback through video, resonating with Sam's emphasis on using visual storytelling to engage stakeholders.
• How AI Will Influence UX in the Next Five Years: 3 Usability Experts Weigh In: This article features expert opinions on AI's future impact on UX, providing context to the themes discussed in the interview.