CXL Institute CRO Minidegree Review | Week 5 of 12
Ali Q Naqvi
I build AI Tools backed by Behavioral Science to solve Business Problems | Ex. Ogilvy UK & Dentsu Aegis
Week 5 Overview
I did not cover a lot of new material this week. I have covered a lot of ground in the first 4 weeks of this course. While it feels good to have completed so many courses and passed the quizzes, I got the sense this week that if I keep moving at the pace that I have been, I run the danger of forgetting what I learned in the first few weeks.
I felt it made sense to use this week to solidify some of the concepts I learned in the first couple of weeks. As I went through this process, I got confirmation that this was the right approach as I truly had forgotten some of the concepts from weeks 1-2 (that is about all I was able to cover alongside the new course I completed this week).
This week’s update will look a bit different to the usual format- I will review the ‘Statistics Fundamentals for Testing’ course as normal- however, this will be followed by a recap of my revision for weeks 1-2.
Course: Statistics Fundamentals for Testing
Instructor: Ben Labay (https://www.dhirubhai.net/in/ben-labay/)
Total time: 25 mins
Difficulty/Skill Level: Intermediate
Key Takeaways
As the title suggests, the focus of this course is on the fundamentals of statistics. This course covers a number of foundational statistics terms that I learned in high school. However, the refresher was much appreciated and I definitely picked up a couple of new things. We covered terms such as:
· Mean- This is also known as the average. For conversion rates, it’s the number of events multiplied by the probability of success (n*p).
· Variance- this is the average variability of our data. The higher the variability, the less precise the mean will be as a predictor of an individual data point.
· Sampling- For datasets where it is impossible/impractical to analyse the entire data set (most cases where statistical analysis is actually useful), we must select a sample data set from the master set and then draw inference about the master set from our analysis of the sample.
· Statistical Significance- Statistical significance is a major quantifier in null-hypothesis statistical testing. A low significance level means that there is a big chance that your “winner” is not a real winner. Insignificant results carry a larger risk of false positives (known as Type I errors). However, we must be careful not to consider statistical significance to be a stopping rule- we must give the test a set amount of time/set a specific sample size before stopping a test as statistical significance can fluctuate significantly during a test- optimisers often make the mistake of calling a test too early.
· p-values- The p-value is the probability of seeing a particular result (or greater one) from zero, assuming that the null hypothesis is true.
· Statistical Power- This is the probability of seeing an effect when it does actually exist, i.e. the sensitivity of your test.
· Confidence intervals and margin of errors- Confidence intervals are the amount of error allowed in an A/B test—a measure of the reliability of an estimate
· Regression to the mean- A regression to the mean is “the phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement.”
· Segmenting- While segmenting isn’t necessarily a statistics-specific concept, it is important to remember that most actionable/valuable insights from A/B testing will often come from segmenting your sample into key segments and seeing how they react differently to your stimulus (variations).
Recap: Key Lessons from Weeks 1-2
· A typical CRO process involves the following steps:
1. Set objectives
2. Ask all the relevant business questions and determine what you would like to find out through data
3. Data gathering
4. Derive Insights from the Data
5. Identify the issues/problems that need to be resolved in the UX
6. Hypothesise test solutions to the problems
7. Design the solutions in wireframes
8. Technical integration and create a UAT version of the test page
9. Launch the A/B test
10. Assess test results to determine which tests won/lost
11. Analyse test results for deeper insights/key learnings/new hypotheses
12. Document learnings
· Conversion rate on its own is not a particularly useful metric- it could easily be gamed by lowering prices / getting your mum to buy something on your website, etc. It is business growth that we are most interested in.
· Best Practices should simply serve as a starting point for your hypotheses about potential solutions to the problems you are trying to solve for a particular website/app. The rest of your process needs to be driven by the data for the specific case you are working on.
· One of our top goals as optimisers is to reduce friction in the customer journey.
· The 5 key principles of persuasive design are:
1. Clarity above all
2. Visual appeal
3. Strong visual hierarchy
4. Conserve attention at all costs
5. One action per screen, when they’re ready
· Visual hierarchy is one of the most important principles behind effective web design. It’s the order in which the human eye perceives what it sees.
· The six steps of an effective copywriting process are:
1. Research: customer, product and competition.
2. Outline and guideposts.
3. Draft copy.
4. Conversion boost.
5. Revise, rearrange.
6. Test.
· Tests have shown that 79% of people don’t read, they just skim. However, 16% read everything. Those 16% are your main target group, the most interested people. If people are not interested in what you are selling, it doesn’t matter how long or short your sales copy is. If they are interested, you should give them as much information as possible.
· A good value proposition answers the following questions:
o What benefit your product provides
o What target audience it serves
o Why you and not someone else
· Every metric that you report on needs to pass the ‘so what?’ test. E.g. “Our blog pages have the highest exit rates on our websites.” “So what?”. “We need to make it easier for people to find more blogs/pages on our website from blogposts so that they spend more time on our site”. “Why does that matter?”. “We know that there is a strong correlation between spending more time on our website and spending more money with us.” While correlation is not necessarily causation, this example shows the types of questions we must ask of each metric we are reporting on and more importantly using to inform our optimisation efforts.
· Averages lie. Instead, look at the data by segments.
· Message-mining is the process of scouring the internet (or other sources if available) for instances of your target customer/audience voicing what they care about the most when it comes to your product/solution. A good way to structure your message-mining process is by breaking down the messages you are ‘swiping’ into the three categories that fit nicely into MECLabs’ Conversion Sequence Heuristic framework: Motivations, Value and Anxieties.
· 5 great Visitor survey questions:
1. Which of these best describes you? (Product Awareness Level)
2. What do you currently use to {accomplish task | solve problem}? (Purchase Prompt Context | Motivation)
3. Is there anything you dislike/want to change about how you currently {accomplish task | solve problem}? (Pain Points | Motivation)
4. What matters most to you when choosing a {product category} like {product/service name}? (Dealbreaker need | Value)
5. Is there anything holding you back from trying {product/service name}? (Objections | Anxiety)
· You are trying to answer the following (potential) customer questions with your value proposition:
1. What’s in it for me?
2. Why should I choose you over (other options)?
Wrap Up of Week 5
What I Loved
· It was great to get a refresher on core statistics concepts- especially with a very practical and direct connection to CRO
· I found the revision process to be very helpful- it is definitely something I will be doing a lot more of as I go through the extensive material in this course.
What Could Be Better
· There were a lot of really important, foundational concepts covered within the Statistics Fundamentals course. However, at times the videos felt a bit rushed. I would like the videos to include a few more examples to help the learner grasp the concept. I had to rewatch the videos a few times- however, this may just be my lack of math skills on show!
Plan for Week 6
I would like to complete the following courses:
· Statistics for A/B Testing
· A/B Testing Mastery
· Advanced Experimentation Analysis
About this series
On 23rd March, 2020, in quarantine indefinitely due to the UK’s (perfectly reasonable) nationwide lockdown, I was lucky enough to gain a scholarship to the CXL Institute’s CRO Minidegree program for 12 weeks.
I am writing a weekly update of my progress through the program. You can find my previous updates below:
Week 1: https://www.dhirubhai.net/pulse/cxl-institute-cro-minidegree-review-week-1-12-ali-qasim-naqvi/
Week 2: https://www.dhirubhai.net/pulse/cxl-institute-cro-minidegree-review-week-2-12-ali-qasim-naqvi/
The CXL Institute CRO Minidegree Program
Conversion Rate Optimisation (CRO) refers to the art and science of converting website/app visitors into customers.
The CXL Institute Conversion Optimisation Minidegree is an online training program designed to be “the most thorough CRO training in the world”. It is taught by CXL Institute’s in-house staff (including its founder Peep Laja) as well as a collection of the leading marketing practitioners in the business.
In its own words, CXL Institute describes the key objectives for students of this course as being to “learn how to convert your traffic into customers, run more successful experiments, and get the skills to deliver consistent revenue gains on any website you work on.”
The approximate total time for this course is: 72 hrs 48 mins.
You can find more information here: https://cxl.com/institute/programs/conversion-optimization/
Thanks for taking an interest in my learning experience. I look forward to sharing more next week!
Ali