How to Master A/B testing?

Everyone wishes to grow their business but only a few people can and others fail over time.

Have you ever thought why does that happen?

If someone wants to reach from point A to B he needs to take a number of steps to reach there.

Simple Right ....

What if you cannot walk to that point?

What if you need to get a car to get there?

What if you don't have a car?

What if that destination is really far you might need to get there via a flight?

What if you don't have enough money?

Somehow you will make the effort, understand the challenges and solutions after doing all that you will reach that point.

Just like that

If you want to consistently grow anything you need to understand how you can consistently improve it.

No business is simple and in whichever environment you are, the market dynamics will keep on changing and as a business, you will have challenges for which you need solutions. You cannot find solutions without experimenting.

Not all experiments will be successful. You will fail more times than you will be successful.

You need to constantly test and learn from your experiments.

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” Jeff Bezos

In order to reach a point in the digital world you cannot leap forward from one point to another point. You will have to improve and optimize your business over time.

Basic Framework for A/B Testing:

·       Research

This is where most of the people fail and fail miserably. In my previous posts, I have explained why is research important and how to collect information for your research.

·       Optimize

You will obviously implement whatever solution you will find from your research and actually optimize it. You will have to do it in phases to understand the impact of that change.

o  Lean Deployment

o  Real Deployment

·       Deploy

Check for negative impacts if there are any.

So Do you actually know when should you run and how many experiments you should run?

You need to check the ROAR Model that will help you to answer this question.

The vertical axis s conversions per month and the horizontal axis time span.

Whenever you will start any business you will start from 0. You might see some dips in sales too but that is not reflected in this example.

The above model is divided into 4 broad phases.

1. Risk (R)
2. Optimization (0)
3. Automation (A)
4. Re-Think

If you have less than 1000 conversions per month you should not run A/B Testing.

No one will stop you but you will not be able to find a winner in those tests. Even if you will find a winner you won't have enough data to be confident from statistical terms am not talking about your confidence here.

I am talking about statistical significance here.

Your conversions can be (Just listing my personal favourite here)

• Leads
• Transactions
• Downloads

Yes, it can also click on a certain action but eventually, you are optimizing for a macro-goal.

Here are some micro-goals that you could try to achieve:

• Reduce your bounce rate
• Increase your average time spent on page
• Convert your readers into customers or subscribers
• Convert your social media followers into customers
• Decrease your cart abandonment rate
• Reduce your page loading time
• Improve your landing pages performance

Check the below chart and when you will reach the 10,000 conversion stage at that time you can do 4 A/B tests per week that are 200 per year and so on.

So Why do we have 1000 and 10,000 checks here for us?

You can check the A/B test calculator from the below link which can guide you throughout the process.

https://abtestguide.com/abtestsize/

The above calculator will help you identify how many days should I run the test and you can clearly gauge that by understanding the number of conversions and number of days it will take you to select a winner from the A/B test.

You should calculate yourself what uplift you are looking for and how many days the test will take to test whether there is a winner or not.

For instance, in the above example, I just added the conversion rate of one of my sites page and I added the expected improvement I am aiming for from it and the number of visitors that will visit the test page.

The calculator will then tell you how many days will the test run.

Well, you need to be smart enough to give full-week runs and check for any abnormal factors and factor that in before jumping on to the conversions. You will have to closely monitor the performance of the page to understand what is happening.

Below are the results that the testing calculator gave;

AB test duration

Minimum test duration of 2.97 weeks

Round up to an AB-test period of 3 weeks (discrete number of business cycles)

Minimum improvement needed

Minimum Conversion rate B of 8.69 % needed to run AB-test in max 4 weeks, with 10000 unique visitors per week on your page

This amounts to a minimum relative improvement of 9% needed in order to run the test at the required levels of power and confidence.

Coming back to the ROAR Model

If your monthly conversions are below 1000 then you are in risk zone and you cannot take more risks at that time.

But when you are risk phase you can make changes but you will not have enough data to validate your tests.

Once you surpass 10,000 conversions per month you will need more resources to help you grow and optimize.

This is when it will become DNA of any company when you move towards automation and there will be a culture of A/B testing.

How can you do proper research?

You can use the below framework for proper A/B testing research and it’s called the 6V model.

So the 6V model, Yes every word starts with V

1. Value — company and customer goals
2. View — web-analytics and web-behaviour data
3. Versus — competitors
4. Voice — surveys, customer support, feedback
5. Verified — scientific research, insights
6. Validated — insights validated in previous testing and analyses

So after completing your research you need to develop a hypothesis that you will test.

How to set a hypothesis?

Your hypothesis will be the base of your experiment so you need set the tone here. With this, you will get your team aligned with a particular aspect you want to test. Everyone will be aligned on what you will be doing, why you will be doing it, and what can be the proposed outcome.

Your hypothesis should follow this framework:

If I apply this (approach), then this behavioural change (data) will happen, among this group (data), because of this reason (approach).

If you want more details about A/B testing you should check out Tom Wesseling's course on CXL for more details.

I will add more details in my next blogs on A/B testing.

Weekly Posts

Week 1

What is Growth Hacking for Beginners? - Detailed Review

Week 2 (1/2)

What is User-Centric Marketing and Why is it Important today?

Week 3 (2/2)

How can you increase the Conversion Rate of your website?

Week 4

Conversion Research Essentials - What should you look for?

Week 5

How to work on your Digital Measurement Strategy?

Week 6

Optimizing Conversion Rate using Google Analytics