A|B Testing with Analytics is the Game Changer : Review

Hey LinkedIn folks I’m back with another post of my series of CXL Growth Marketing Mini Degree. So let’s start from where we have left:

I realized that I’m adding too many details and I’m quite lazy to type all of that therefore I’m adding just the overview now. Running test is very important for the overall success. But testing without analytics will end up in to a big disaster. Hence lets jump into the various analysis which can be conducted in order to make your campaign a success and aggress towards the North Star Goal.

Mouse tracking: Track people’s mouse movements to see where are they clicking and how far are they scrolling.

Watch the replays of the user sessions to understand where exactly are people facing issues.

Google analytics health check-up: Check the following stuff:

·????????Profile setup ( configuration and admin )

·????????Filtering of traffic ( agency, office, data cleaning )

·????????Goal and funnel configuration ( key reports )

·????????Code review ( On-page analytics code )

·????????Page and process instrumentation (funnel, steps, forms)

·????????Any issues that would prevent insights.

All of this information is covered in an insanely detailed manner in the course covering what, why, and how exactly to do that.

Let’s move to the next course which is AB testing mastery. This is a highly theoretical course so let’s cover the basics here:

The biggest mistakes in A/B testing are:

·????????Experimenting on not enough data.

·????????Not using proper statistics.

·????????Not knowing if an experiment fell it or not, not really know when to call a winner.

·????????Not understanding what to experiment on.

When to use A/B testing:

·????????When you deploy something on your website.

·????????Can be used for research and it can be split into more parts.

Check abtestguide.com

Power: When you start trying to test on pages with high power (>80%) otherwise you don’t detect effects when there is an effect to be detected. (False negatives)

Significance: When you start trying to test against a high enough significance level (90%) otherwise you’ll declare winners when in reality there isn’t an effect. (False positives)

The 6V conversion canvas includes value, versus, view, voice, validated, and verified.

Create behavioural segments:

Typical e-commerce flow examples:

·????????All users on your website with enough time to take action.

·????????All users on your website with at least some interaction.

·????????All users on your website with heavy interaction.

·????????All users on your website with clear intent to buy.

·????????All users on your website that are willing to buy.

·????????All users on your website that succeed in buying

·????????All users on your website that return with the intent to buy more.

How to calculate the lengths of your A/B tests:

Use the AB test guide calculator to calculate the lengths of the AB test. Conversion rate can be different in fay or nights, weekday or weekends. Even different times of the day can influence people buying behaviour.

AB test should run between 1–4 weeks because if the test runs longer, there is a chance that people will delete that cookie and may land in a different variation of the website which will make all the data useless.

Monitor your AB tests and run experiment dashboards. Stop an AB test when something is broken, If there is an SRM error, or If it’s causing too much loss of money.

AB Test Outcomes:

Before you start analysing:

·????????Know the test duration, only check during the test that conversion is coming in and the traffic is evenly distributed.

·????????Determine how to isolate the test population (users who have seen the variation)

·????????Determine the test goals and how to isolate those users (users who have seen the variation followed by a test goal )

When we start analysing:

·????????Analyse in the analytics tool and hot in the test tool.

·????????Avoid sampling.

·????????Analyse users, not sessions.

·????????Analyse users who have converted not users and total conversions.

·????????Check if the population of users that have seen the test are about the same per variation (SRM error checks)

Inconclusive result

No winner: It doesn’t mean your hypothesis or design was terrible. We are just unable to prove that it is outperforming the default.

If the result can be implemented: The chances of a negative impact are quite small the variation has a higher measured conversion rate than the default.

Does it add to the button line: No

Should I filter out the users that could not have been influenced: Yes

Should I Digg in all sort of segments to find winners: No

It’s a winner: Implement ASP

Statistics fundamentals for testing:

Sampling:?Populations, parameters, and statistics

·????????The population is the pool of all potential users or people in a group of things that we want to measure.

·????????What are the examples of population parameters?

·????????Sample statistics are used to make inferences on the population parameters.

Mean, Variance, and standard deviation:

·????????Mean is the most common measure of central tendency.

·????????How spread out the data is what’s known as a common variance.

·????????The common measure of variability in statistics is the standard deviation.

·????????The common measure of variability in statistics is the standard deviation.

·????????Standard deviation is the measure of variance.

·????????The shape of the spread, or the amount of variance, has a direct impact on the sample size needed to compare the two means.

Confidence Intervals:

Confidence intervals are the range of values, defined in a way that there’s a specific probability of the value of that parameter that lies within.

Confidence interval includes:

·????????Mean

·????????Sample

·????????Variability

·????????Confidence level

·????????Confidence intervals are the amount of error allowed in AB testing. It is the measure of the reliability of the estimate.

·????????True conversion rate can’t be measured.

Statistical significance and the p-value:

·????????Statistical significance helps us quantify whether a result is likely due to chance.

·????????The P-value is the probability o obtaining the difference you saw from a sample if there really isn’t a difference for all the users.

·????????P-value does not tell us that the probability of B>A

·????????Does not tell us the probability that we will make a mistake in selecting B over A.

So folks that’s all for this week. There is lot more to cover in my series therefore I will try to add fewer details from now and cover more courses in my blogs.

Stay tuned for next week’s update.