Why A/B testing in Email always falls short & what to do about it

I've concluded that most A/B testing over email marketing is a waste of time.

Yes, hot take, but hear me out for a second: this is based on working with close to 200 DTC brands over email (see my recommended A/B tests).

Most A/B testing is generally implemented on the front end of the website to convert first-time visitors into buyers.

Experimentation holds credence here as it can plug the gap in a leaky bucket leading to tens of thousands of incremental revenue gains.

With email, that's not the case, and there's a very specific reason why.

Your customers already know who you are!

You don't need to run so many experiments to "convince" them to buy from you again.

You just need to stay top-of-mind with engaging content.

After working with close to 200 brands and auditing in the region of nearly 500, guess how many times I've come across a brand with a consistent - and effective - A/B testing protocol in place over email that drives incremental revenue?

ZERO.

I'm not kidding. There have been some decent efforts, but the juice is seldom worth the squeeze.

There's a fair few reasons for this:

1. As aforementioned, subscribers already know who you are, so implementing tests to convince them to buy again is often wasted efforts - you should spend more time being creative in your marketing instead
2. Most D2C brands lack the repurchase rates anyway to reach statistical significance with their tests (less than 30% of consumers buy again for most brands)
3. Sample sizes are usually too small to derive meaningful learnings from for most brands

?

These are the core reasons why A/B testing fail consistently over email to generate anything meaningful.

?

I'm including Holdout Tests over campaigns as well as sending time optimisation in this bucket as well.

?

Both these strategies lead to transient findings that seldom hold over the long-term and are influenced by constant variables which distort their credibility.

?

Bold statements, I know, but it's backed up by enough research at this stage of my career to make the claim.

?

With that being said, am I against ALL testing and diminishing the impact a testing culture can have?

?

Absolutely not. I do believe in testing 1 thing, relentlessly.

?

Pop-Ups!

There are only two ways you can influence the results of your clients:

?

1. Drive people back to the website to buy again
2. Get more people on the list to buy again

Pop-ups are unique in that they not only drive number 2 but also have a direct influence on converting first-time visitors into buyers, which is where the majority of efforts should be focused with A/B testing.

?

Pop-ups tend to fall under the jurisdiction of email marketers as their responsibility and they're a key lever you should pull to drive conversions and build up the brand's database.

?

There are a huge amount of variables you can test with pop-ups across multiple parts of the customer journey making testing a non-stop treasure trove of incremental gains.

?

A few (obvious) ideas I recommend testing monthly:

?

• Offer (always number 1)
• Zero-party data capture
• Micro-commitments
• Timing
• Page scroll

?

Test your pop-ups relentlessly with the focus on driving conversions and increasing list growth and keep a diligent log on your findings.

?

It's the main focus area that will unlock leverage for you as an email marketer and genuinely move the needle.

WATCH & LEARN ??

Are you prepared for the BIG changes coming to email deliverability in February 2024? Watch this video to understand the basics quickly.

The biggest misconception around email deliverability? It's this.