Never Shop for A/B Testing Hypotheses!
Adetola Julius
CRO Specialist | UX Researcher | Creating a delightful customer experience that helps e-commerce brands generate more revenue from existing traffic while turning first-time buyers into loyal brand advocates
Does anyone happen to have a list of possible hypotheses? And of course, is okay sharing it with me?
Sorry, I didn't ask the above question! Someone in a closed group I belong did.
And as a Conversion Optimizer, I was startled!
Why so?
That's because the questioner is looking for what to test (multivariate or A/B Test) but went about looking for the answer in the wrong place.?
Now, one of the reasons CRO gets a bad rap from certain quarters is largely as a result of the amateurish approach of Conversion Optimization practitioners like this is.
The most important part of A/B testing is knowing what to test! We could test all kinds of things all day along and all year round. But if they have no business impact, we would have merely wasted time and money.
It is very easy to fall into that trap of shopping for hypotheses to test. But don't do this!!
Testing is a cost. Even if you're using free testing tools, there are other costs you should be aware of:
If you waste precious time testing useless things, you're obviously going to get useless results.
A/B Testing is a problem-solving adventure. To embark on that adventure i.e. designing a solution that solves the problem, you need to ask yourself:
"What problem am I looking to solve?"
Yes, it is a conversion rate problem, no doubt. But what exactly is the problem?
What problems on my website are responsible for my low conversion rate? Where are the problems located?
The difference between tests that win and tests that are inconclusive is your hypothesis.
Shopping for a list of possible hypotheses is only courting failure and frustration. These hypotheses may have absolutely nothing to do with what's going on your website. So, testing them is, at best, a guessing game.
领英推荐
The hypotheses that you need are and that will be useful to you for a long time to come are those that are derived from the conversion research done on your own website. Not the one from another website!
With Conversion Research, you are able to gather data and uncover insights about what's going on your website. For instance, Funnel Visualization in Google Analytics may show you that people are dropping off big time at the "Add-To-Cart to Checkout page.
That's a huge problem you need to fix, right? That's the “WHAT”!
Google Analytics is great at telling you the "what". It gives you the quantitative data. But it doesn’t tell you the “WHY”. It does not tell you why people are dropping off so much at this stage in the customer journey.
You could try to remedy that problem through your Abandoned Cart Email sequence.?But if? you really want to unravel what’s responsible for this huge drop off, you need to gather quantitative data - surveys, interviews, heatmaps etc.
Let me tell a little story here. I've been working with a business owner for a while now. She balks at the mention of doing customer interviews. What are her fears?
She dreads the thought of hearing negative things her customers would say about her brand. I insisted this wasn't even negotiable. We just needed to do the interviews. Eventually, she agreed.
This weekend, she handed over to me 15 of those interview transcripts. I have only read 3 of them but the insights I have gained about her customers from those 3 alone are priceless!
Those insights will definitely come in handy when implementing subsequent changes on her website. We trust those to improve customer experience on the website significantly, and by implication, conversion rate.
That’s how invaluable qualitative data is. You will get insights to complement issues identified in Google Analytics.
Now, only the data from your Conversion Research should form the basis of your hypotheses. Not some random hypotheses floating around the internet. Once you have your data, it's now time to create your hypothesis. Thereafter, you can prioritize what to test before running any A/B testing.
How do you prioritize your hypotheses? That's a topic for another day.
What I have said above can be summarized in the following steps:
In conclusion, please don't go down the rabbit hole of hypotheses shopping and random A/B testing. And don't do it for clients. Get your hypotheses from your own conversion research!
Consulting, Technology Assurance, Billing Integrity, and Quality Management System Professional
3 年Nice.