28 Testing Methods That Are Not A/B Testing

Dear Product Person,

Today I'm thankful for all the ways you can test that are not A/B tests.

Executives and product people think A/B testing is the only thing on the testing menu. ???

For me, it doesn't usually make sense to A/B test.

Here's why:

1. A/B tests should happen as late as possible. They might be the most scientific approach, but they require a lot of traffic. Plus, they're usually live and in code, so everything must be "launch-ready." Even if it fails, stakeholders don't like to "relaunch" ideas, so they stay launched regardless of the results.
2. Who needs a strategic designer when you're optimizing a page? Where should the button be put to get the most clicks? It would be better to hire a marketer or follow best practices.
3. I prefer the more ambiguous and challenging upfront side of product work—the innovation or product discovery side—because it's just more exciting.

Luckily for me (and for you if you're bored by A/B tests), there is a whole ??cornucopia?? of testing methods. Yay! You don't have to limit your menu to just A/B testing anymore. ??

A Framework for Testing Products

Here's a tasty little framework that lives rent-free in my head.

This helps me know what type of test is needed.

Neat, right? You can work through these risks in any order but often go left to right.

I won't go into details but check out an example based on the Tesla Cybertruck here.

Ok, let's use the framework to test.

Get ready to skim because this section is a lot! ?

7 Ways to Test a Product Idea for Viability

Viability is about ensuring the product meets financial and market goals in the long term.

Here are some ways to test for viability risks:

1. Revenue Modeling: use spreadsheets and pricing estimates to understand which combinations of offerings will appeal to users
2. Competitive Analysis: examine relevant competitors and their positions in the marketplace to determine whether your idea is worth pursuing.
3. Customer Interviews: interview your target user and uncover problems, pain points, and issues that will tell you whether your idea is solving a real problem
4. Willingness-to-Pay Surveys: use a survey methodology like Van Westendorp Price Sensitivity Meter to determine an acceptable range of pricing based on user feedback
5. Break-Even Analysis: figure out the bare minimum in revenue needed to cover the costs and make the idea profitable
6. Channel Effectiveness Analysis: research acquisition costs to determine the marketing channels with the cheapest CAC (Customer Acquisition Cost) to make your idea more viable
7. Jobs-to-be-Done Analysis: interview users to determine the “job” they hired you to do to establish the long-term viability of a problem and solution

7 Ways to Test a Product Idea for Feasibility

Now, it’s time to look at how to test the technical buildability of the idea.

Here are some ways to test for feasibility:

1. Capability Gap Assessment: evaluate the execution teams’ expertise to determine if it matches the requirements of the innovation project
2. Functional MVP: build an early prototype to test the minimum required functions to establish the technologies needed to make the project a success
3. Scalability Tests: create sample workloads to test the architecture of your planned tech stack, ensuring the finished idea can withstand the stress of daily challenges.
4. Compatibility Tests: test your chosen tech stack against other tools and libraries that will be needed to implement the innovation idea
5. Pilot Program: run a test of the entire experience with a small user group to see if you can handle the operations load of the innovation idea
6. Interactive Prototype: build a clickable prototype to uncover any UX/UI bottlenecks early in the process
7. Workflow Efficiency Analysis: create user flows to map out the potential bottlenecks in the steps that the user will take to accomplish core functions and try to simplify those steps

7 Ways to Test a Product Idea for Desirability

Desirability is extremely important, especially if you don’t know your customer well. But don’t worry—there are many methods you can use to test desirability.

Here are some of the most popular ways to test for desirability risks:

1. User Interviews: interview your target user and uncover needs, preferences, and desires that will tell you whether your idea will be desirable through deep user insights.
2. User Surveys: like interviews, user surveys query your users, allowing you to ask the same questions to a larger group with shallower insights.
3. Preference Testing: show multiple product versions and have users pick their favorite. This one can be an early concept version that can later be A/B tested.
4. Prototype Review: share your early prototypes and gauge your users’ initial responses by ranking features as desirable. This can even be paired with a user interview.
5. Landing Page Tests: design a page that outlines the core value you’re trying to test, and have users either sign up for a waitlist or interact in a way that shows interest through page data
6. Video Explainer Tests: create short videos to pique and measure interest in innovative ideas.
7. Social Listening: track hashtags or influencers to analyze sentiment related to innovation ideas.

7 Ways to Test a Product Idea for Usability

Last, we have the usability testing approaches designers are keen to use.

Here is a list of testing methods for usability risks:

1. Task-Based Test: give potential users tasks that mirror the functionality of the final product and measure their performance
2. Tree Testing: test the information architecture of the idea by having the tester complete tasks on a simplified, text-based version of your innovation project to see if they can find what they need
3. Heatmaps: use tools for tracking where and how often users click in certain areas of a page to determine where users linger or fail on a page
4. First-Click Testing: check to see if users know where to click to accomplish tasks associated with the core functionality
5. Heuristics Evaluation: use specific heuristics, such as those provided by Jakob Nielsen, to determine whether your idea meets core usability principles like error prevention and user control.
6. Feedback Collection: monitor customer support tickets and/or provide feedback forms when the innovation idea is launched to catch usability gaps quickly
7. Post-Launch Analytics: track specific metrics such as feature adoption and session durations to determine if the product has usability bottlenecks

A/B testing might be the turkey at the table ??, but the cornucopia of other tests makes it a balanced meal. ??????????????

What to do now

Share the full version of this article with your boss so they realize all the ways that you can help them test.

To learn more about how to set up a test with any of these methods, use my Experiment Cards.

Until next time!

-Jeff Humble

P.S. I'm curious how many of these you've already done. Let me know down in the comments if you've used one of these methods!

#innovationtesting #testingmethods #productinnovation #abtesting #feasibility #viability #desirability #usability

This article was originally published on Jeff's blog. Read the full version here

Get Jeff's newsletter as an email + memes, events, and news every two weeks with the Fountain Institute Newsletter. Sign up here.

Book Jeff for an innovation project by booking a free consultation call here