10 Tips to Avoid Common Product Experimentation Pitfalls

This is Part 2 of the series on experimentation. In the last part, we discussed; when you should run experiments, what are MVTs and how to define success:

---

Product experiments help product teams get to know the customers better, reduce risk due to uncertainty and connect actions to the results in a methodical way. Additionally, by running a high velocity of experiments with a mix of small and large bets and staying objective during the design and analysis of experiments, they are able to create a long lasting competitive advantage.?

But that is not to say that running high-velocity impactful experiments is easy or obvious. Here is a list of 10 tips that might not be obvious:?

1.?Set your hypothesis upfront with specific metrics-?This has a few advantages. 1- It makes it very clear to everyone involved what metrics you are trying to influence and aligns the team to that objective. 2- Since it’s a hypothesis you have already inherently accepted that there is a chance for failure, so you avoid the reputation risk and the pressure that many teams feel to deliver successful features. 3. You become less likely to get derailed by changes in metrics that are not primary to the experiment.

2.?Make sure there is enough tracking upfront-?There is nothing worse than waiting a few weeks just to realize the metrics you were reporting are not accurately tracked. This often requires your developer to have a good idea of the context of the experiment, which metrics would be affected and how.

3.?Estimate how long the experiment will run before you start-?In order to properly engage stakeholders and ensure other business changes are not going to impact your results, you will need to have an approximate timeline. To understand how long an experiment will run you will need to estimate the approximate change expected and how quickly you can get to that sample size (the rest is just statistics).

4.?Define which counter metrics you will watch out for-?Before you start your experiment you also want to define other business metrics that may be indirectly impacted by your experiment. This is crucial if you are in a complex business and you are experimenting with high risk ideas. These counter metrics ensure you and your stakeholders that your experiments are not having negative effects.

5.?Experiment in phases gradually increasing risk and reward-?New and novel ideas are by definition high risk, high reward. In order to reduce exposure; start with smaller components of your hypothesis or smaller changes to the experience. If the building blocks are successful you can slowly build in more functionality and complexity as well as iterating where required.

6.?Design for high velocity testing-?To increase the impact of your tests, design for high velocity of experiments. If experiments succeed, double down and optimize further and if they fail try something new.

7.?Triangulate findings with qualitative user data-?Sometimes it is hard to fully understand why certain experiments succeed or fail. This is where deep understanding of users and qualitative research is the missing piece of the puzzle. By layering experiment results with user research and sympathy you can learn insights that will not be unlocked by experimentation alone.

8.?Customize your experiments for different personas-?Often, changes applied to user experience across your full user base are not statistically significant as a whole. Only some of your users will respond positively and if they are a small proportion of your base, you will not see a significant change. Instead analyze your experiments for each segment separately and consider which segments will require their own experimentation idea backlog.

9.?Keep customers in the same treatment-?To avoid incoherence in the customer journey and contaminating your data, you can attach an experiment identifier to each of your users which ensures that they do not see multiple variants of the same experiment. Most commercially available experimentation platforms have this functionality out-of-the-box.

10.?Take into account the external changes-?Many times your competitors’ actions or overall market conditions can change how your users react to a certain experience. A common example is seasonal trends, users behave very differently in signing up for self-improvement programs in January vs the rest of the year. If you have a continuous experimentation pipeline, keep in mind which experiments may be vulnerable to these changes and consider retesting at a different condition.

This is part 2 of a series on Product experimentation and Growth. In the next article, I will cover “ How to design high impact experiments?"