Growth startup (5/12): mastering a/B tests review

Time to take another step in this series of growth marketing, based on my learnings from CXL’s Growth Marketing minidegree.

Last Sunday, we reviewed some pillars of conversion research:

• Selecting a framework (the LIFT model, Web Arts framework and Invesp Conversion Framework)
• diagnosis steps,
• heuristic analysis,
• site walkthroughs,
• letting your visitors and consumers be protagonists.

GROWTH STARTUP SERIES PREVIOUS POSTS:

Growth startup (1/12): open-minded growth vs. traditional marketing review

Growth startup (2/12): building a customer-centric growth process review

Growth startup (3/12): running growth experiments with research and testing review

Growth startup (4/12): conversion research review

Now, we’ll review something very, very popular in optimization:

a/B testing. ;)

A/B tests from the beginning

I'm not an optimization expert. And the customers I work with do not have a formal optimization structure in their companies, despite realizing that they need to optimize and their sincere desire to take action.

This leads us all, both me and my customers, to discussions that are often well-intentioned, but that in fact are not related to strategic optimization experiments.

For this reason, these discussions about A/B testing are limited to a certain superficiality. While we discuss issues such as experimenting with a CTA's color or changing fields on a form to increase clicks, these are not really strategic optimization issues.

Optimization for business results

A/B tests for clicks and behaviors are perhaps the two main uses of this experiment method that professionals and companies just starting out in this area think about.

It is a natural act - and for survival in some situations. If I need more visitors clicking in a CTA, why not take an A/B test? Likewise, why not change the layout of the email to encourage people to open more links?

But if the intention is to actually generate results for the business, the discussions should revolve around higher issues, such as:

• Transactions
• Revenue per user
• Potential lifetime value

See the difference?

"But please move away from the level of clicks and maybe shift behavior, but please add transactions. You want to drive business, you want to grow faster, and have a whole team and put A/B testing in the DNA of your company."

Ton Wesseling, the founder of Online Dialogue

More clicks and behavior change are fundamental. Nothing against these goals. However, optimization in your company should (ASAP) gravitate towards more massive stars, such as transactions, revenue per user and potential lifetime value.

But how to not cannibalize?

Try to imagine the following situation: you are marketing to encourage your leads and customers to spend more money buying the product/service your team manages. In the meantime, another team in your company works just as hard as you do to make customers spend less money for a better retirement.

We have a conflict: you want to optimize to increase the average spend of leads and customers. On the other hand, the other team works for them to turn their money into savings, reducing their mood for spending money on services/products like the ones you offer.

But why talk about all this if our focus is on A/B testing?

If the intention is really to have a recurring optimization program for the business (including A/B tests) with more mature objectives in addition to increasing clicks and changing behaviors, then it is necessary to decide very clearly what are the metrics and goals of each team.

That is, how each team will work for:

• Transactions
• Revenue per user
• Potential lifetime value
• Etc.

Research, research and research

We have already talked about the importance of research for any optimization initiative.

It has to be that way because we have also seen here that optimizing is not playing dice: it is working with hypotheses that must be tested through experiments that generate results that are proven to be reliable.

That is why it is important to start from models that help in the organization and the research process so that your A/B tests (and your optimization strategy as a whole) result in truly strategic insights.

Wesseling has a model called the 6V Conversion Canvas:

• Value: what are the company's values? Which objectives and focus have the greatest impact on the business?
• Versus: what competitor analysis and best market practices can you find?
• Voice: what insights can you have from your consumers and customers (surveys, customer service, sales team, etc.)?
• View: what insights can you have from your analytics and user behaviors on your website, blog, landing pages, etc.?
• Validated: what insights have already been validated in other optimization experiments?
• Verified: what scientific research, insights and models are available on the market?

How to optimize optimization efforts?

After researching and gaining valuable insights, the next step is to begin your A/B testing journey.

But when to start exactly? And when to invest even more in it?

Wesseling also has interesting answers to these questions. See the image below:

In this image, we see a clear distinction between risk, optimization, optimization automation and re-imagining. And the touchstone here is conversions.

"There's a border between risk and optimization. And the rule of thumb border to me is 1,000 conversions per month. If you are below 1,000 conversions per month, you cannot run A/B tests. Yes, you can still deploy it, you can still run A/B tests, but it's really hard to find a winner because you are just too low on data and even if you find a winner, chances are pretty high this is not a real winner. You are measuring this as a win but in reality, it's not a win."

Ton Wesseling, the founder of Online Dialogue

As you can see, A/B testing is much more than trying to increase clicks on a button or CTA. They are an invaluable resource for possible increased results for the entire organization.

That is why they are related to discussions and plans that must be more profound.

EXTRA MILE

grow your growth marketing

A/B Test Size Calculator

A/B Test Calculator

A/B Test Duration Calculator