Experimentation: The Growth Accelerator

One thing that if a company can learn to do right can accelerate their growth by many folds is running effective experiments. It is a widely discussed and used methodology yet widely misunderstood for start-ups.

For growth-stage companies experiments have become a part of everyday life with the marketing team running experiments, the product team running experiments, the engineering team running experiments to optimize the app platform, and so on so forth. Because of this in the end we don’t get concrete results because the same person is being treated by Marketing, Product, and Engineering. It is hard to understand what is the efficacy of each experiment and what were the results. So in most cases, experiments are run but are inconclusive because of many reasons the above being one of them.

However, the truth is that experimentation is the key to growth for start-ups. If a company is able to establish the right experimentation strategy then it will be able to grow in a more accelerated manner with known drivers.

What is experimentation?

To understand what is Experimentation and how is it different than Analytics let’s understand how we capture data for an organization. So, basically, there are two ways in which you get data.?

Let’s understand each with an example of a freemium app that is built for dating. Obviously, growth is going to be one of the key metrics here. In a freemium app, growth is a function of the number of free people and the number of paid subscribers. Now, assume that 1% of the overall base is paid. If we were to figure out who within the free group looks most alike as a paid user, we will have to figure this out using historical data by assessing the behavior of paid and free users this kind of model is commonly known as look-alike modeling, this is done using historical data. This will fall under the category of Analytics.

To understand Experimentation let’s say, the free users get to view only five profiles unless they convert to paid, and then they can view more. Once they have viewed five profiles and they want to view the sixth one they come across a paid subscription nudge, this clause is called an upsell hook. Let’s say for this company the upsell hook is set to purchase an annual plan. But the company also has another plan as an option a monthly plan. Now a product manager might have a question “What is the best-paid plan to present for this upsell hook?”. The product manager may hypothesize and say “Hey annual plan requires a significant investment. So in the upsell, we should be pitching a monthly plan.”? Now this idea is new and there is no historical data available for this. In this case, we will run an experiment and capture enough data to figure out with significance which plan is better for the company to promote.

How does experimentation drive growth?

Knowing how to optimize the user experience and the flows through a series of experiments clarify the organizations’ business path. So, it is advisable for growth-stage companies to experiment often. But, the challenge lies in doing it the right way. Unfortunately, it is fairly uncommon to see experiments designed and executed well.?

Essentially, it is like the same person being treated in ‘N’ different ways. Therefore it is not a clean sample and we often don’t get very good conclusive results.?

Why Experiment?

There are two major reasons why companies should experiment

1. Hypothesis Testing: This we already looked at if someone has a strong hypothesis like in the above case the product manager had a strong hypothesis that “if we change the hook from annual to monthly, we are going to increase our trial rate by 10%.” This sounds reasonable because people would not want to invest a lot
2. To prove Causality - For understanding the other reason for experimentation, let's assume the dating app we took as an example has a predictor of paid subscription model where you know in between day 10 and day 30 if they do an activity, let’s say the activity is connecting their Spotify playlist to their profile, you establish that if they connect their Spotify playlist, they have a 90% chance for subscribing to a paid plan using the predictor of paid sub-model. In this case, you are establishing a high correlation between the event and the subscription. You want to establish a causality to make a core function change within your user behavior to insert the playlist connection. This means running an experiment and confirming people who are likely to be paid subscribers are actually doing this event.

We now understood that the first reason for experimentation is purely hypothesis testing and the second reason is to establish causation. For almost everything else, your analytics would be sufficient. Now we only talked about the mess you get because one person gets treatment from many many departments.? However, there are many many other reasons why experiments fail. We are going to talk about that in the next post.?

Until then happy experimentation!!!