How do you choose the right sample size and duration for your A/B tests?

Sample size is crucial in A/B testing. A small sample size may not yield statistically significant results, while a large sample size may waste resources. Factors such as baseline conversion rate, effect size, statistical power, significance level, variability, and cost should be considered when determining the sample size. Higher baseline conversion rates require smaller sample sizes, while larger effect sizes require smaller samples. Higher statistical power necessitates larger samples. Higher variability requires a larger sample size to detect a significant difference. Finally, a larger sample size may be more expensive, but it may lead to more reliable results.

To make your results statistically significant you need your sample to be adequate and the duration long enough to conclude that there is a relation between the variables you are testing. At minimum you need 30 users and 14 days. Keep testing.

The thing I see more people make mistakes on is the stimulus. One page has a blue background, some text, 2-3 images, several headlines and subheads. One page has a white background, some different text, 2-3 different images, several different headlines and subheads. More people engaged with the page with the white background-ergo white backgrounds are better. Or maybe the text, headline, subheads and images were better. A/B test need to compare small difference, not wildly different pages. Amazon runs hundred of small tests on pages every day. Traffic is statistically significant. They changed a button, swapped a picture, put a link in a different location. If you just compare 2 different pages, you don't know what caused the preference.

My summary: (1) use the smallest possible sample size that can result in statistical significance (2) hold the experiment as long as possible without creating operational complexity and risk.

Many people think of A/B testing as a way to gain a last bit of optimization in a pipeline, or squeeze a few more % of conversion out of a campaign. Don't forget that you can also A/B test early with drastically different variables. Here's a use case: A client is convinced they have their target audience nailed down, but your research reveals untapped markets. Propose they A/B test their campaign. It allows objective data to triumph over subjective hunches, better results for all!

From my experience: Keep as many variables the same as the original campaign you are trying to test. Ideally there should be only 1 unknown variable to have a successful A/B test. In order for the A/B test duration to be sufficient, check and see how long the main campaign took to gain consistent results. Use that timeframe as a benchmark for the duration of your “B” test. Once you have clear results such as higher conversion rate, higher CTR etc, make the decisions and implement the best strategy.