Why A/B testing falls short, part 5: The visitor that bounced - You know nothing, Jon Snow

This is part 5 of a 5-part series highlighting challenges you’re likely facing in your A/B testing strategy. Check out the first 4 parts of this series:

Part 1 You’re experimenting with your customers

Part 2 You don’t understand WHY!

Part 3 You need to wait a looong time

Part 4 You’re optimizing for the wrong persona

Problem #5: The visitor that bounced - You know nothing, Jon Snow

According to Wordstream’s founder Larry Kim, the average cross-industry conversion rate is 2.35%. Seems reasonable. Some industries have lower averages, and some higher, but without a doubt, the overwhelming majority of any website’s visitors do not convert.

So why is it that the entire focus of A/B testing is on the success cases? In other words, why are we stuck on a strategy that simply chooses the webpage that converts 3% over the one that converts 2%?

Wouldn’t it make more sense to focus on understanding the failed cases? To focus on reducing the 97-98% that are leaving our website without taking any action? Elephant in the room??

The reason is actually quite simple and unfortunate. We know very little, if anything at all, about the people who do not convert.

Let’s take a step back: How does A/B testing inform us about our audience?

It doesn’t.

After spending the better part of an hour going through Optimizely’s website, it’s safe to conclude that the focus is on being an experimentation platform, not an insights tool. 

That’s great. The value props do make perfect sense:

“Enable everyone to improve every web experience”

“Run more experiments”

“Web experiments that load in the blink of an eye”

All good.

So where’s the bit about understanding the customer? Isn’t that a must-have in order to be better informed so that we can actually improve?

Experiments are great, after we’re informed

Alright, let’s get scientific. Wikipedia says the following: “An experiment is a procedure carried out to support, refute, or validate a hypothesis.”

How are we creating those hypotheses?

In her blog post ‘True Confessions of a Digital Marketer: “I Go With My Gut”’, Janet Muto describes what we do most of the time: We guess!

It’s not our fault. 

We don’t have the time nor the tools to ask our audience the questions. So we go with what we do have. Our gut. As Muto explains, “Your gut is specific to you. Your gut has good and bad days (more kombucha, less burritos, okay?) Most important: your gut is not the actual thoughts and feelings of your target audience.”

We need the opportunity to ask our website visitors questions.

What, who, how & why?

A/B testing tools, and certainly true if we expand the definition to all tools analyzing live traffic data, show us what people were doing on our website.

They don’t tell us who those people were.

They don’t tell us how conclusions and decisions were made.

And they definitely do not explain why (see part 2 of this series: You don’t understand WHY!).

Why people came to our site.

Why people stayed for 2 minutes and 35 seconds and then … left.

Or even why people purchased our product or service.

How could we possibly learn and improve without knowing why?

Get to know your audience: Uncover why people don’t take action

The goal is to create the most effective ad campaigns. The most effective email marketing and the most effective website. Produce the highest conversion rates and deliver to our business’ top line. To increase revenue!

A culture of experimentation is necessary for success and there are most definitely questions that cannot be answered unless we test them. But we have to be smart about how we do it.

According to eMarketer most A/B tests fail to deliver results.

We’ve covered many of the causes and the price we pay for it is now clear.

We need to change our behavior.

We need to listen to our customers and seek insights pre-testing. Before going live!

The only way to run successful A/B tests is to uncover why people don’t take action, before going live.

A theory can be proved by experiment; but no path leads from experiment to the birth of a theory - Albert Einstein

Tomer Azenkot is the Chief Revenue Officer at WEVO.

Please follow @Tomer and @WEVO on LinkedIn to hear more ideas on how to improve digital experiences.

About WEVO:

WEVO is the first technology platform that optimizes digital experiences before you go live. Leveraging crowdsourced visitor insight and artificial intelligence, WEVO generates recommendations that have proven to significantly increase conversion. WEVO has successful experience working with Wells Fargo, Fidelity Investments, Blue Cross Blue Shield, Harvard University, Intuit and others.