3 Points for A/B Testing
In the 1920s, what started as somewhat of a taste test with cups of tea turned into the start of how we start to think about and consider A/B testing strategies. Ronald Fisher began designing the standards of how we run scientific experiments, by understanding controls, statistical significance and experimental design.
Fast-forward a hundred years, and brands lean on the same foundation Fisher explored with tea and milk, by leveraging a scientific model to improve customer experiences from direct mail to mobile app journeys.
#1 Define the Hypothesis?
What are you hoping to learn from the experiment? Identifying the issue, potential solution with expected outcome and reasoning allows for you to define the null and alternative hypotheses.?
Alternative: Administering a change and/or a variation to an existing scenario
Null: No differences will be observed from a test and control scenario
An example for today’s digital marketer:
Alternative: By reducing the options during checkout, customer CR% increases.
Null: Reducing options during checkout will not affect CR%.
领英推荐
#2 Time and Size Matter
A lot of marketers will talk about statistical significance within testing. This is important because with statistical significance, you can know the results were not left to chance. But, of course there’s a small caveat. Statistical significance can be strong … or weak.?
Don't worry. This is not a math lesson. To learn more about statistical significance and p-value (and why it’s important), check out a few resources here and here (or ask your local Data Scientist!).?
No matter how far you dive in, understand that a larger sample size (go bigger than 100, if you can) and testing for at least 2-3 weeks will allow your A/B test to kickoff on the right foot.
#3 Try, Try, Try Again
A/B testing, by nature, is not a one-and-done or a set-it-and-forget-it strategy but rather a way to continue exploring new ways in which customers engage with your brand.
By adopting a strategic testing methodology, you continue to learn more about your customers with each interaction. And by refining those processes, customer experience and satisfaction follows. Or, at least that is part of the goal (I think I hear another test…).