A winner or an insignificant result? A/B Testing outcomes

Hello there! In today’s article, I’ll be talking about A/B testing outcomes, what does it mean if you have an inconclusive or significant result and the difference between Bayesian and frequentist statistics.

Talking about your outcomes, is the test a winner or not? Understanding when your A/B-test is a winner, and what to do when it's not a winner. Before you start analyzing, you have got to:

        i.           Know the test duration. Only check during the test the conversions are coming in and the traffic is evenly distributed. We have to be sure that we know when the experiments started if we are looking at data, and also when it ended. So we can check if the data is there for A and B, but also to get the data out of the system.

      ii.           Determine how to isolate the test population (users who have seen the variation). If this is only on new users? Was this experiment run just on new users? Or was it run on the full population, if so, then we will need to analyze new users? And we need to know how to do this. And also the same for the goals. What was the specific goal, and how do we set up measurement of this specific goal gets the right data out. Of course, we really need to analyze this A/B experimentation in our own solution

    iii.           Determine the test goals and how to isolate those users (users who have seen the variation followed by a test goal).

If you have an inconclusive result, what does it mean;

No winner? Does this mean that your hypothesis or design was terrible? The answer is no, it doesn’t mean your hypothesis or design was terrible. You were just not able to prove that it's outperforming default. It could even be that you created is in reality better than the default. And in this measurement, it didn't come out as a win, it could be a false negative. You just don't know. So, your design wasn't terrible.

Can the results still be implemented? Yes!, especially, if you are on the right sites. So, there is an uplift, but uplift is not enough to have a significant result. The chances of having a negative impact on your business case are quite small. So if you were just checking it for a deployment, if we are going to release this out to see if it works, it should not have a negative impact. It's a different measurement you're using. You want to measure for sure, if it's inconclusive or positive, and those are different calculations. But here, if outcome is inclusive, you can still release the codes, it probably has no impact. But that is what you were looking for.

Also, does it add to the bottom line if you implement this one? No!, this result was not significant, So you were not able to prove that B was out-performing A. So no, nothing to add the bottom line. Should I filter out users that could not have been influenced? Yes. If you run the experiment on the whole population, but a part of the population is leaving the page after like one second or two seconds, you just filter those out. They should be the same for A and B because if that population is like four times as high in the B variation, then probably something is wrong with your experiment, it's broken. Don't look at the results, throw the results away. Look for a better bottom and start all over again. But if it's the same, you can just leave them out, and the conversion rate of both variations will go up, it will be easier to find the significant results. So, filter out users that could not have been influenced by your experiment.  Should you dig in all sorts of segments to find winners? No, you're going to get inconclusive results.

If you have a significant result, what does it mean;

It’s a winner. Implement ASAP since it’s a winner. The conversions will go up. If you measured a certain % uplift, will conversions go up that same %? No, you have a measured outcome and you have the reality and you don't know the reality. It could be the reality is even bigger, it could to be even smaller and there's a way bigger chance of being smaller. After many tests, the average win percentage is almost true but right skewed due to Type-M errors and can you can finally calculate how many Type-1 errors there are.

Bayesian vs Frequentist Statistics

The new hypothesis is telling you the conversation rates of default and variations, so original one, challenger, are the same. The alternative is that variation B, your challenger, is better. And you can run an A/B experiment, collect all the data, and then look at the P-value. Could the data plausibly have happened by chance if the null hypothesis is true? And so, if the conversion rates are the same, you will see an uplift, but if it can happen by chance, then we still go for the new hypothesis. It happened by chance, yes, we fail to reject the new hypothesis, so conversion rates are the same, inconclusive result, or no, we're going to reject the new hypothesis, and we'll say we have a significant, better variation, B. It's quite hard to understand. If you're in a really mature company, it makes sense.

So, what's the alternative? That's going from frequentist to Bayesian. This is what happens, in a Bayesian test evaluation, and the same results, 1,200 for conversion A,1,260 in B, looks like this. There's an 89% chance of B outperforming A. A Bayesian approach makes sense if you're focusing on direct values, for just optimization. If you want to learn about user behavior and doing research experiments, then it makes more sense to take the frequentist approach. Also from frequentist statistics, to really prove an hypothesis you need more than one A/B experiment. You need several winners, really building up a case for this main hypothesis, and to be considered to be really, really true. But that's more on the research sides of things. That's the difference between frequentist statistics and Bayesian statistics.

That’s it for this article, I will be posting more articles in the following weeks with more knowledge and insight about Growth Marketing.

If you want to learn more in detail about growth marketing or any other marketing course, feel free to visit CXL Institute website. They have a wide range of marketing courses and top 1% professionals in different fields of marketing that impact first class knowledge. You can also apply for their mini-degree scholarship programs just like i did.

Catch you later!