Product Thinking Playbook – Wizard of Oz Testing

By Chloe Blanchard

Continuous discovery and product testing are precursors to building better products. As a rule, the sooner you start validating or invalidating your ideas, the faster you can begin to mitigate risk and save time and money while identifying new opportunities. And while there are numerous ways to test products and gain user insights, few are as effective at simulating the intended experience as a prototype.??

Undoubtedly, the contextual data and user responses you can elicit from prototyping are second to none. Unfortunately, the challenge is that it inherently requires significant time and money, primarily the two things you’re hoping to mitigate through testing in the first place.?

But wouldn’t it be nice if you could test and validate an idea, and assess desirability and usability without actually investing the time and cost to develop the product or features?

You can – with Wizard of Oz Testing. Heel clicking optional

What is Wizard of Oz Testing?

Like its namesake, Wizard of Oz (WoZ) Testing is a prototyping technique where participants interact with a simulated product experience that they believe to be functioning and autonomous. Much like the man behind the curtain was the “wizard,” to imitate an immersive experience, this technique relies on an unseen operator completing the more complex or responsive tasks that haven’t been developed yet.

Why would product teams do it?

When product teams can simulate experiences with users, they can more accurately and thoroughly validate critical hypotheses related to the desirability and usability of product concepts without having to commit time and resources to build the product or feature.?

WoZ is most useful when the product needs to process incoming contextual data (e.g. the user’s time or location), manual inputs that impact the flow, and/or output custom responses (e.g. responding to a request). Voice experiences are a common candidate for this method, as it would be difficult for participants to imagine the value of a voice experience if faced with unsuccessful interactions. This allows you to meet the participant’s baseline expectations and ensure more fruitful data is obtained.

When should product teams use it?

Prototyping: Typically designed for testing with participants, this technique can also be used to demonstrate a product idea. Testing is conducted to understand the desirability better and/or usability of an experience before building. Desirability is the primary motivator for conducting Wizard of Oz Testing. Usability testing using Wizard of Oz prototypes should be focused on identifying high-level principles for how users want to interact with the product such as validating the type of interaction that should be used.

Concept Evaluation: The need to simulate, evaluate, and prioritize a list of potential features is of course valuable, but also, quite costly. When assessing the desirability of one or multiple product concepts or critical features, WoZ can be incredibly helpful to mitigate that cost and still get the data you want.?

Demand Validation: Testing a product idea in the participant’s context/environment helps determine problem/solution fit while amassing qualitative data to help validate or invalidate your desirability hypothesis.

Who is required?

Design Researcher (Lead): Responsible for defining research objectives and determining when and how to implement Wizard of Oz testing. Also, they contribute to the prototype to ensure it aligns with research objectives. They lead the testing sessions and analysis.??

Product Designer (Lead or Support): Explores how to simulate the desired experience for testing using minimal technical implementation and design/build the prototype so it’s ready for testing. They also contribute to testing sessions and analysis.?

Product Manager/Strategist (Support): Defines research objectives and supervises the prototype design and build. In addition, they support the testing process, from implementation to analysis.?

Software Engineer (Support – if needed): Explores how to best simulate the desired experience for validation. Also responsible for building the prototype for testing and contributing to post-event analysis.*

*We’ve even been recently successful at utilizing this technique without any coding or reliance on software engineers.

How do I do it? (Best Practices)

1. Let the fidelity of your prototype be determined by the research objectives and build feasibility

Low-fidelity WoZ Prototypes typically are perceived as early-stage, and most of the functionality might not be present; the participant should expect some imperfections across the experience. Only feedback on the concept, not the design, should be invited. The intent isn’t to be used to assess usability.

High-fidelity WoZ Prototypes require considerable time and effort, impacting the overall timeline but providing a better experience, increasing the chances of the participant interacting naturally with it. In addition, testing with a high-fidelity prototype can yield more feedback related to usability and design decisions. The flip side of using high-fidelity prototypes is that participants may hesitate to give critical feedback on the idea at a high level if it is perceived that a great deal of time and effort has already been invested.

2. When it comes to Wizard of Oz Testing, test the test!

Due to the nature of Wizard of Oz Testing, it is critical to pilot these testing sessions to ensure that the prototype and control room is functioning as expected. Identify risks or problem areas where prototypes or participants might get stuck, and make sure you have a backup plan for getting back on track.

3. Don’t lie; just don’t disclose everything

Wizard of Oz Testing is not about lying to participants that the product they’re testing is fully functional; it would be unethical to lie outright about the experience in which someone is participating. Instead, the researcher presents limited information and relies on participants to make assumptions about the product they are interacting with. If a participant asks questions about the prototype, use your best judgment regarding how much to reveal, but it is highly discouraged to fabricate information explicitly.

The Product Thinking Playbook is filled with tactics and techniques that help product teams build better products. Click?here?to download your copy of the complete playbook, and stay tuned as we share more from it in the coming weeks.