A-B Testing CXL Review 7

?

(Hager Khaled –Article #7)

?

I am Hager Khaled a junior copywriter looking forward to be a media Buyer so that I enrolled in Growth Marketing Minidegree from?CXL Institute?in order to understand what is Growth Hacking.?

I have signed up for the Growth Marketing Midigree from CXL Institute?, and fortunately, I have granted a free scholarship. Therefore, I will be writing an article weekly, summarizing the lessons and specifically what I have learnt and understood.

Our ship will sail this week with AB Testing, let's begin

What is the Purpose of AB Testing?

A/B Test is designed to compare between two versions of e-mails in order to know which one performs better such as converting more leads, more traffic, more clicks and so on

We change some elements such as the line and color of CTA and the e-mail subject lines.

A/B Test?is important hence choosing the perfect version will greatly affect the??engagement of customers and certainly the target persona.

A\B testing can be used for research, lean deployment and real deployment. A-b test can be considered as a validation step

On the research phase, we are looking for signals or impacts or flat lines needs to be done.?

On the one hand, Lean deployment is based on finding winners or in other words how the present action will affect the future goals. On the other hand, Real deployment refers to finding no negative signals.

In A-b testing, we do not use KPI, but we use goal metric The goal metrics are Clicks, behavior, Transactions, Revenue per user, Potential lifetime behavior.

The most important goal metric to heavily rely on is transactions. Compared to Revenue per user, Clicks is not a great goal metric as it could not correlate with purchases

Word of caution, is A/B test tools and most calculators in the markets are only compatible with binary variables. So you are converting, or you are not converting.

There is a transaction, or there is no transaction.Lay your focus on 0\1 binary outcome

Overall Evaluation criterion (OEC)

A good OEC should not be short-term focused (e.g., clicks); to the contrary, it should include factors that predict long-term goals, such as predicted lifetime value and repeat visits." So, ideally, an OEC is based on metrics in a short-term experiment that are good predictors of long-term value.

What is a Null Hypothesis?

?

The null hypothesis is a typical statistical theory which suggests that no statistical relationship and significance exists in a set of given single observed variable, between two sets of observed data and measured phenomena.

While, Statistical power is the likelihood that an experiment will detect an effect when there is an effect to be detected

?

What?does false positive means?

?

A false negative is a Type-II error. This means that in reality you have created something that is better than the default, but you were not able to detect this in an A/B-test. You were not able to reject the null hypothesis (why in reality you should have done this).

?

?

?

?

?

The relationship between false positive, Power and significance.

False positive is all to do with significance levels. False negatives has to do with power levels, if your power is to low, you will find too many false negatives. So you think all your experiments doesn't make an impact, when you measure it but in reality.it could be lots of them are making an impact but you are just too low on data to be able to detect these ones.

?

You were just not able to detect the impact with your A/B experiment; it could still be there is a positive impact. So don't throw away ideas just based on one AB test it can be a wrong thing to do.

?

?

All about power and significance thumb rule of thumb with power when you start A/B testing try to test pages with a high power, like 80 percent or more so you have a big chance of finding a winner if there is a winner to be detected, otherwise you don't detect effects when there is an effect to be detected.

Too many false negatives so 80 percent or higher.

Significance, also when you start, try to test against a high enough significance level, like 90 percent otherwise you'll declare winners when in reality there isn't an effect, false positives.

Potential, what is the chance of the hypothesis being true? Impact, where would this hypothesis have a bigger effect? And then power, the chance of finding a significant outcome. And easy, how easy is it to test and implement?

Research with the 6V model

A view, voice, validated, verified and value is in there

View:- Is the customer low, middle or high?

?Voice: Does it match with the voice of the customer information we have?

Validated :- Do we have past experiments that indeed, confirm that this hypothesis is true? Is there back up from science?

Verified:- Is there verified data that this hypothesis could be true? And how strong is the match with the KPs we're trying to optimize, with what the stuff, the company's trying to reach for.

?

?

?

?

?

?

?

?

Roar framework

Now the names of these phases are risk for phase one, phase two is optimization, phase three is automation, and phase four is the re-think phase, especially if growth declines or if it goes down. So risk, optimization, automation, re-think. his is why it's called the ROAR model.

?

?

Important Notes

·??????You should not shorten the length of your A/B experiment. You will call winners who are not winners, who are what we call false positives.

·??????If you have below 1,000 conversions per month, don't do A/B testing. It will be very hard to find a winner.

?

?

?

There are many prioritization models out there.???The most two well-known are PIE and ICE. They're almost the same. It's like potential, importance, and ease, or impact, confidence, and effort. How easy to do, how much effort does this cost? What is the potential? What is the impact? That's kind of same. And how important? How much confidence do I have that this will work? All these models are roughly the same.

?

?

?

?

See you next week in a new journey, Stay tuned.

?

?

?