"Don't think, test." — Faulty assumptions in experimentation

There is no more damaging (and pervasive) idea in product strategy. The lure of “just test it” has captured a generation of product leaders into thinking that shipping quickly and measuring will eventually lead you to a useful product and successful business. If you’re in that camp, you’re making a major mistake and I want to show you the limits of this idea.

I get the allure of this idea. You’re sitting in an endless meeting with designers, marketers, and product managers. Everyone has an opinion about the best direction. Saying “let's just test it” seems like an exit from the conversation. No one has to be wrong, no one has to be overruled. You're not letting the most forceful or most senior voices bulldoze good ideas. Just let data tell you which one is “better.” And if it doesn’t work, roll it back and say “it was just an experiment.” Everyone wins!

Wellllll… no. An experiment is not a way to tell you if a product is good or bad. You think you're just measuring "the product" in the experiment and if the experiment results are positive on your key metrics, you have a "good" product on your hands. But you are making many assumptions along the way:

• The short-term performance of the product during your experiment is representative of long-term performance. You don’t actually care about “improved engagement over four weeks.” That’s what you’re measuring. But what matters is something long-term. Maybe lifetime value. Do you understand the relationship between four week changes and “lifetime?” Usually, no. And in my experience the relationships are surprising. Short-term effects (good or bad) usually revert to null long-term effects. Null short-term effects can occasionally turn into positive long-term effects.?
• Your current users are representative of future users. All your experimental results have a secret caveat – “assuming our future users look like our current users.” Just because a feature performs well with people who currently use your product is no promise it’s good for new users or different kinds of users. So depending on experiments to evaluate a product will tend to bias you towards what works for your most engaged current users. If your user expectations change, your user acquisition strategy changes … the product you’ve experimented your way into may not be good for them.
• You have a way to measure "success." You may have a “north star” metric, but it’s usually not the actual output of your business. You may fantasize about something like “incremental long term revenue.” But it’s the rare team that has a trustworthy version of that metric available for experimentation. You’re probably focused instead on conversion, or some kind of retention on some horizon. But it’s not the actual outcome of the whole business. So an experiment being “positive” does not in a general sense mean “good for the company.” It just means “caused some number to move.” It's entirely possible to improve conversion in ways that does not represent an overall good outcome for the business. No one number can tell you if a product is good.
• You will get a statistically significant result. The dirty secret of experimentation is that the median result is null. In other words, no detectable effect. What then? Is the product good or bad? Should you ship it? Time to think.

You can mitigate these issues with smart experiment design, clever statistical techniques, and good culture around using data to make decisions. But it’s by no means simple. You have to think. There is no such thing as a product strategy without a deep understanding of the people, the product, and how they fit together. Experimentation can shine a light on specific aspects of that relationship. But a "guess and check" approach to building a product that people value is at best inefficient and at worst leads you into mediocre products.