Bayesian A/B testing: A practical experimentation model for marketing

For businesses, effective decision making is the goal. Statistical significance and other fluff are just supporting actors...

I recently led my organization's transition from traditional frequentist testing approach to Bayesian testing. This successful transition brought great benefits in terms of flexibility, real-time decision-making and intuitive results. This article explains some of the intricacies of Bayesian testing and why it is better than the traditional approach.

Understanding Bayesian A/B Testing

Bayesian A/B testing is rooted in Bayesian statistics, a branch of mathematics that updates the probability of a hypothesis as new evidence becomes available. Unlike traditional, or frequentist, methods that rely on fixed p-values and long-term averages, Bayesian testing gives marketers a clear probability that one version is better than another based on the data at hand. In e-commerce, where decisions often need to be made quickly, this ability to update results dynamically makes Bayesian methods particularly attractive.

Why It Matters for E-Commerce Marketing Teams

1. Faster and More Accurate Insights

Traditional A/B tests often require large sample sizes and long testing periods before producing statistically significant results. For an e-commerce business running multiple campaigns at once, waiting weeks or even months for results can be costly. Bayesian A/B testing, on the other hand, allows marketers to start making informed decisions with smaller data sets. This can speed up testing cycles and allow teams to pivot quickly based on real-time data.

For example, a marketing team testing two versions of a product landing page may be able to determine, within a few days, that version A is 80% likely to outperform version B. This is much faster than waiting for a traditional test to hit statistical significance.

2. Intuitive Decision-Making

Bayesian A/B testing translates complex statistical outputs into understandable probabilities. Instead of relying on technical jargon like p-values, Bayesian methods provide straightforward answers like, "There’s an 85% chance that version A will generate more sales than version B." For marketing teams, this simplifies the decision-making process. Stakeholders can act with greater confidence, reducing the friction often associated with interpreting traditional A/B test results.

3. Optimizing for Revenue, Not Just Clicks

In e-commerce, the ultimate goal of any marketing effort is revenue generation. Traditional A/B testing often focuses on click-through rates or conversion percentages, which may not tell the full story. Bayesian A/B testing, however, allows for testing with more nuanced business metrics. For example, a test can be designed to predict the expected lifetime value (LTV) of customers based on different email campaigns. This makes it easier to optimize for long-term profitability rather than just immediate clicks or conversions.

4. Flexibility in Multi-Variant Testing

E-commerce teams often want to test more than two variations at once. With traditional methods, this can be cumbersome due to the increased sample size and complexity of analysis. Bayesian A/B testing can handle multiple variations more efficiently, making it a better fit for testing things like product recommendations, discount strategies, or personalized user experiences.

Bayesian Testing Foundations

I will intentionally leave all the technical details out, but the core principle of Bayesian testing is Bayesian inference, which uses prior beliefs and new data to update the posterior probability. The formula for this is:

Posterior = Likelihood x Prior / Evidence

In practice:

• Prior: Your initial assumption about the success rate of a variant (can be based on historical data or a neutral guess).
• Likelihood: The observed data from your test (e.g., clicks, conversions).
• Posterior: The updated probability that a particular variant is the best after considering the new data.

Example: Real-World Bayesian A/B Test for Campaign landing page.

Let’s imagine you are testing two variants of campaign landing page:

• Variant A: Landing page with images, text and CTA buttons.
• Variant B: Landing page with just text and CTA buttons and no images.

Our hypothesis is that we expect a clean landing page with no images to convert better and have more CTA clickthrough.

After 5,000 visits:

• Variant A: 1,000 clickthrough (20% open rate)
• Variant B: 1,100 clickthrough (22% open rate)

With Bayesian inference, after this data collection, you find:

• Variant A has a 30% probability of being better than Variant B.
• Variant B has a 70% probability of being better than Variant A.

Given this data, you can confidently conclude that Variant B is more likely to be effective, and you can begin to deploy it while continuing to gather data for further confirmation.

So far it sounds very similar to what frequentist method could achieve, but following scenarios can all be accommodated in the model to make it relevant and flexible to true nature of business needs -

• Define priors - In past campaigns, Variant A had a 20% clickthrough rate, so you can start with this assumption.
• Adjust strength of priors - Suppose the head of marketing with many years of experience running such campaigns, tells you he believes very strongly that the variant without images (B) won’t perform any differently than the original variant. You can account for this by increasing the strength of the prior. This will now require much more evidence to change our beliefs.
• Have different priors for control and variant - The lead designer sees your results and insists that there’s no way that variant B should perform better with no images. She feels that you should assume the conversion rate for variant B is closer to 15 percent than 20?percent. Rather than using one prior to change our beliefs, we want to use two—one that reflects the original prior we had for A and one that reflects the lead designer’s belief in B.

If this approach is better, why its still not widely adopted?

Unlike frequentist plug and play approach, this method requires human intelligence and business awareness to conduct successful testing. There are plenty of frequentist 'calculators' available out there but Bayesian testing requires deeper understanding of Bayesian Statistics especially if custom models are to be built. That is where human intervention kicks in.

Likewise, business awareness is important mainly due to setting up of priors. These priors are most effective if they represent true sense of business and adapts to its direction. If you are new to Bayesian testing, deciding on the priors can be tricky. However, starting with non-informative priors can mitigate this.

You will find that some of the niche testing software providers have either widely shifted to Bayesian or provide it as one of the methods to test. However, popular vendors still resist this method. That is because it shifts from their narrative of 'marketer friendly automatic toolkit'. Frequentist methods are automatic yes, but I would vote Bayesian testing as more marketer friendly for sure.

Conclusion

For e-commerce marketing teams, the ability to make fast, data-driven decisions is critical for success. Bayesian A/B testing offers a more agile and intuitive approach compared to traditional testing methods, providing faster insights, clearer probabilities, and the ability to optimize based on meaningful business outcomes like revenue and customer lifetime value. By adopting Bayesian A/B testing, marketing teams can make smarter, faster decisions that directly impact the bottom line, giving them a competitive edge in the crowded e-commerce landscape.