Dose #65: What to A/B Test on Emails

Matt here with your weekly Subscription Prescription ??

?

Dose #65: What to A/B Test on Emails

Building on last week’s issue on upcoming order emails, in this dose, I explore what you should be testing (or not) on these emails. We’ll specifically dive into some offers you can try and the overall reasons why testing makes sense.

Always be testing. It’s every marketer's worst nightmare or path to glory. Spend time testing the wrong things, and you can do more than just waste time or money. Test the right things, and you can unlock more revenue and personal glory.

In past doses, I’ve offered some insight into how you should approach testing overall, focusing on product pages to increase conversions or offers to boost retention (check out Dose #49).

If you’re still starting out (less than 250 subs or less than 1,000 site visitors per day), spend time honing in on better acquisition tactics and improving offers like bundles or upsells. Don’t overcomplicate what is most likely a lean team managing too many things already!

In fact, I would double down on data collection. Give away samples, discounts, or better offers for feedback from subscribers. You can do this with post-purchase survey tools (right away), or on order notification emails. Qualitative data will give you stronger insights into how you can grow than testing subject lines on emails.

That said, I would spend some time ensuring upcoming order notifications are on brand, have strong copy, and include upsells to popular products (see last week’s dose for more on this).

To test effectiveness, we need a large enough audience for two reasons. The first is so that we can test with statistical significance (great article on that here). The second is so that any results are worth the time it will take to make them happen. (Is it worth an hour a week to manage a test that drives 5 more orders a month?)

So, what should you test on order emails?

If I’m going to test anything on an upcoming order notification, it’s going to be an offer. Can I incentivize someone to buy more? Can I do something to keep them from canceling? These are the things to think about when testing with emails.

Let’s use a case study I saw recently from StayAI about a popular brand, Obvi, as an example. StayAI makes it easy to A/B test offers on emails. You can create offers yourself or use their AI platform to create tests for you. It’s pretty cool.

Obvi faced a pretty common problem: high churn in the first few months of a subscription. They wanted to come up with an incentive to keep people around longer, so they started notifying customers that they had a free gift coming up in their next order.

The goal was to determine if a free gift would get people to stay on the subscription instead of pausing/skipping. It worked really well for Obvi and resulted in 85% higher retention than customers who were not notified about the gift.

Interestingly, they found that the free gift offer was really only effective on the 2nd month. It had little effect on the 3rd month and no effect on the 4th month.

What else should you test?

Here’s a list to keep things simple:

• Copy to improve retention (again, you need about 1,000 people opening an email to see complete results) by framing the value you’re delivering in a way that reminds people why they subscribed and why they should stay subscribed
• Free gift offers versus no free gift to see how this affects churn
• Upsell offers and specials to increase the amount this customer is purchasing from you
• Where you place items in the email, such as the notice on how to pause/manage an order versus showing someone what they’re getting next (can you convert more upsells if it’s higher or lower on the page?)

A note on A/B testing: this can feel pretty easy when you’re testing with a tool like Klaviyo to adjust subject lines, but great testing can often be more effective with better segments.

For example, if you have one really popular product and a lot of other items, I’d start by offering:

• The really popular product to everyone that doesn’t have it already
• Various products as upsells to anyone with the popular product to identify which products drive higher AOV

I would also invest time in identifying customers that are spending more than others. If someone is buying a higher-than-average AOV, I’m really interested in finding out how to get them to buy more and keep them around longer.

Sometimes A/B testing is less about running concurrent tests, and more about identifying the more valuable segments in your business (and the best time to make offers to them).

I feel like this week’s dose rambled a bit as I dove in and out of testing, segments, and offers. Since this can be pretty confusing and sometimes overwhelming, I hope the advice helped, but know that you can always hit reply and gut-check some ideas with me.

Stay tuned for Dose #66 next Tuesday!

?

- Matt Holman ??

Head of Growth, QPilot