Always Be Testing

For those well versed in Testing, none of this will be new to you. However, with the new hype of AI and Machine Learning, traditional testing has been pushed to the sidelines. As I'm working with various teams and determining the needs of the business, I realize there is some basic education that is required. While you might be data rich, some basic knowledge is needed before jumping into the latest technologies.

As a former consultant and analyst, I have always been taught that testing is core to any digital marketing strategy. It’s been refreshing to see these practices seep into other areas of marketing over the years. I see testing as essential to every aspect of marketing.?

Any marketer not testing is dating themself, giving away an easy competitive advantage. I have planned and executed thousands of tests, not only as a direct marketer but as a digital marketer. In this day and age, there aren’t any marketing platforms that don't provide testing options. Adtech includes the usual suspects from Facebook to Google. Martech players from Salesforce, Marketo, and Optimizely–to name just a few–are pretty typical tools used across organizations with testing capabilities.

While the concept of testing is not new, it’s has evolved in the last decade for 2 key reasons:

1. Data is plentiful - as the data has become more available. The method of collection and where consumers are interacting with brands digitally allows more ‘touchpoints’ to collect to obtain marketing ‘signals’
2. Testing is part of almost ALL techstacks now -? the ability to execute tests and obtain results exists and has been more efficient over the last few years.

With the growth of machine learning/neural networks and AI, I think some marketers have become lazy on learning what specially is working. Going back to the basics of testing principles is still needed for marketers.

Having been in both sophisticated and rudimentary testing environments, I’ve realized that these common areas remain true in all scenarios despite the age of data, AI, automation, and technology:

1. What is Testing
2. Why Test
3. Always be Testing (ABT)
4. Impact or die?
5. Marketing Foundations
6. Rigor and Structure?
7. No one escapes testing?
8. KPI driven?
9. Broaden your perspective???
10. Summary and Key Steps to Test

What is Testing

Before I dive into the testing fundamentals, let’s make sure we all agree on what ‘testing’ really means for marketers.

One Concept with Many Terms

Whatever you want to call it, it’s testing

Here's how I've defined it: Testing is a method that takes a sample of your audience to understand how a marketing attribute (channel, offer, creative, etc) has an impact on consumer behavior (awareness, sales, leads, revenue, etc). True testing uses statistical techniques to ensure that you can quantify the impact.

The beauty of a true test-and-learn system means you will know precisely what is working and what is not. It quantitatively proves if something works and–if set up correctly–why it performed so well. For example, you may know that a higher promotion value drives an increased response rate, but you may not be sure as to how much or where the point of diminishing returns will be for that promotion. This is where testing will help you get the detail you need.

Testing reduces your risk by taking a small sample of your audience and trialing something to quantify its impact (a promotion, a creative element, a message, a CTA, etc).

An advantage of digital is the opportunity to sample an unbiased group of customers in a live environment. Users shouldn’t know they are being tested but instead should react to your variables in their natural state as consumers. Other advantages of digital include data, measurement, speed, and the scale of users.

I try to remind teams new to testing that it’s not just about a winning test but also about ‘learning’.

We all hope for the most impactful results, but testing also allows us to learn why something works or does not.

While testing covers a broader perspective, many in the industry (especially those in digital) focus on A/B testing where you create a randomized process that involves showing two or more versions to different segments of website visitors simultaneously. The goal is to determine which version has the maximum impact and drives business metrics. Given the nature of digital, it’s very easy to set up split tests across many channels such as:

Website experience

Paid Advertising

Social Advertising

Mobile apps

Marketing campaigns

Email

Direct Mail

Ecommerce

Push Notifications

Pages, flow, video, landing pages, business model (e.g free shipping, returns, etc) - trade-offs, backend algorithm changes, timing, cadence, new products/services, etc

Why test?

You might be asking “why test?” if we have so many new technologies like AI to do the heavy lifting. For Testing, a lot of the heavy lifting is in the planning and set up? - test cells, sample sizes, tracking codes. While AI is always an option, there are some things to consider:

1. AI? requires a TON of data. Not all organizations naturally collect a lot of data.
2. It’s typically a black box, meaning the machine does the work for you so you likely don’t know what exactly is driving the result. Programmatic Media works this way but it's designed to continually optimize. For other channels, like a website or email, the volume levels are much smaller and you also need to know what's working to modify your efforts on your own (vs relying on a algorithm like digital media)
3. Foundational testing principles ultimately feed into what you learn from AI. Better to start with the basics versus depending on a machine doing all the work for you.

Now let’s go back to why we test as marketers:

Through testing we can…

• Identify ineffective strategies
• Understand their target audience
• Measure the impact of campaigns
• Improve marketing efforts
• See how products and strategies work
• Understand the competition
• Determine what works best for their audience
• Repeatability of results on a full population

The final bullet is critical and sometimes forgotten. Testing is executed on a sample population. The point is you don’t want to risk your full population - so you test on a smaller group. Marketers naturally want to repeat successful results on their full audience population. However, this may not be as intuitive for other groups who are just looking to measure success at the moment of launch.

ABT - Always Be Testing

Recently, we launched this new mantra within my Digital Marketing team. We now start meetings with it and recently got the team hats to start socializing the concept across the organization. If you do it right and follow the testing principles - you get a hat! We are happy to say there seems to be strong momentum across the company.

We’ve created a full ‘Test and Learn’ system that has become an integral part of how we approach all Marketing initiatives now. We launched testing as an all-the-time, everywhere concept. ?

Done right, testing should be ongoing and continuous. It should not be a ‘one and done’ or a ‘once in a while’ project. We suggested that anything you run on the entire customer? base, should initially be set up as a test on a smaller scale. Most companies actually have small pockets of testing going on typically within the digital or CRM teams. However, we can do better! Wherever there is ‘data,’ there should be testing. If there is no data, then create it or find it! To truly maximize your efforts in testing, you must extend it to all parts of the organization

To build a true ‘test and learn’ system across your organization, you must enforce a formal structure where planning and processes can be instituted. I've been in environments where every department claimed they were 'testing' but in reality, they were all just powering through many new efforts without any structure, measurement or consistency.

Unfortunately, many believe they are already testing when they are not. Many might be trying new programs and even launching new efforts but this is not formal testing. Testing requires discipline, knowledge, and planning to maximize your outcomes.

Testing will allow an organization the following:

1. Repeatability: While other departments might only need to justify running an offer, marketers need to guarantee replicability in the hopes of ‘scaling’ the test in a true GTM environment. You want to see that same or better resume when you run it across your entire population.
2. Impact: You need to precisely quantify the impact without the impact of other factors.
3. Precision: You want to make sure you can predict or ensure the impact with numbers.

For example, putting out a new offer is NOT testing if you are not setting it up in the right way to ensure your results are correct. MORE importantly, results must have the ability to be replicated in a scaled environment for all customers.

There’s nothing wrong with trailing things or even measuring something using a pre- versus post- technique. However, if you (like most marketers) need to reproduce these results in a broader context, a formal test structure is required.

Some questions to ask when planning for a test:

• What is the measure of success?
• What are you trying to understand? What offers drive CTR or what drives more Revenue/LTV?
• Will you know at the end of the test what particular variable drove the impact (promotion, copy, creative, audience, channel, etc)?
• Do you have a baseline to compare the test? You might have results, but can you compare them to another test cell that did not use or launch with that particular variable (offer, incentive, product, etc)?

Impact or Die

Be ready to fail a lot before you figure out your winner. The only way to find winner tests is to power through many tests that you eventually find the right marketing mix. The beauty of digital testing is that it can be run on actual consumers/users. There is no artificial group or sample but instead a ‘live’ customer group.?

While the true advantage of digital testing is to allow you to power through many tests efficiently at such a scale, it does not give you a pass to prioritize quantity over quality. If you are running hundreds of tests, make sure you are running tests that are driving impact-otherwise move on. The industry average success rate of AB testing is around 12% to 15% (Optimizely reports that 25% of all AB tests run on their platform are successful). This means you might have to power through 8-10 tests to find a true winner.[footnote]

If you are new at testing and do not have the benefit of historical learnings, a smarter way to approach your testing is to take more traditionally impactful variables. Incentives, promotions, audience, and channel type are good places to start. Creative elements like color, font, message should be your secondary variable list. Do your research, talk to other marketers, colleagues in our industry or do some Google searches to determine what others have seen as powerful variables. Don't test for the sake of testing but instead, figure out smarter tests/variables and challenge the team to drive impactful results.

A famous example is from Microsoft: On Bing there was a test idea that was put on hold and then launched as a test variation that ultimately increased conversion with a 12% lift and $100M in revenue. A simple a/b test could have quickly allowed them to make an impact immediately. What good is ‘thinking’ about a good test if you aren’t going to run it? [footnote]

In summary, keep these things in mind for testing:

1. Drive to a business metric: revenue, lead gen, sales, etc.–be as specific as possible. Make sure your metric can be calculated with the data that you already have.? If you start testing, make sure you have a specific impact based on KPIs (e.g. sales, revenue, profit, click through rates, etc.) Be specific and identify these upfront.
2. Focus on impact and learning: Look for other levers/variables that are more impactful. No need to keep waiting out a test if it’s not working.
3. Speed: This goes with the phrase ‘fail fast’. The benefit of modern testing is that you don’t need to wait a long time to get results. We have the ability for rapid iteration given the volumes and data (particularly in digital) to get to statistical significance.

I worked with one company that minimally ran tests for 12 - 18 months. This was overkill and they prioritized time over minimum sample size. This direct to consumer retailer, could have used the learnings to drive revenue impact 6-9 months earlier.

Testing for All

While testing principles have been used in Direct mail, Customer Relationship Marketing, and digital marketing, they haven’t been used consistently across all of marketing.

One of the practices we have found as part of the testing education is to make ‘Test and Learn’ a ‘must-have’. In other words, we are encouraging teams to test before they do anything in a GTM scenario. How do you know what outcome/impact you will have if you run a certain campaign? With a test you have the ability to forecast the results on your full audience with true confidence.

We have started to see success in educating testing principles to other parts of the organization including revenue management, finance, operations, and IT.

Getting started

Some core principles that ensure reliable results:

1. Clear objectives
2. Controlled Variables
3. Randomness
4. Sample sizes
5. Reliable metrics/data
6. Reproducibility
7. Statistical Significance

We now know the what and why of testing. Let’s talk about how you should structure it.

Remember, testing is about taking a ‘sample’ from your broader population and doing something(s) different with that population.

The beauty of testing is that it is interactive and never really ends. It’s a continuous loop of taking a sample, testing on it, and replacing your full population with what has been working in your testing cells. This forces an Always Be Testing mindset where you are parallel pathing–making an impact (with strong test results)--but also continually learning and optimizing.

You can’t go from 0 to 60 without a strong foundation in data, and statistics, and an understanding of your key marketing levers. Starting small is great but in the digital world companies like Amazon, Google, Spotify, Booking.com, and Netflix all are sophisticated testers due to the nature of their data and being ‘digital’. Don’t be frustrated if you are not at this level. They all started doing very simple testing before they got to the level they are at now. Given their data collection, data capabilities, data scientist, and ability to ‘control’ with their algorithms various test cells, it should not be surprising that they are at the highest level of testing today.

So where are you in your Testing Journey? Do you have a culture of Test and Learn? What will it take to “Always Be Testing” (ABT)?

So with all the hyperbole around AI, ML, and Automation, maybe we go back to basics and create an ABT culture with foundational elements.?

A recent example demonstrating the power of still setting up a proper test was our Media Mix Modeling (MMM) work required more 'variation' in the data across several channels including OOH (Out of Home). This is a really tricky channel given the 'always on' nature of outdoor print (e.g. how do you remove a billboard after it's been launched unless there is another advertiser waiting). As a post mortem for our MMM, we will be designing specific test cells using digital OOH Q1 and Q2 this year. Testing is still necessary for our key learnings.

Final thoughts

In summary, a testing culture allows an organization to:

1. Save money
2. Save time
3. Learn things in a safer less risky environment
4. Replicate the results in a scaled environment

Remember, It is near impossible to ‘overtest’.? On the other hand, It is quite possible to waste a lot of time, energy, money, and resources with poorly set up tests that don’t move your business. I've seen this happen time and time again.

In summary, you want to build an ongoing Test and Learn culture to drive impact, learnings, repeatability and scale.

Resources;

Revenue Driven CMO podcast

My former Boston University student Caroline Amato - thank you!

Tallulah Doeringer for editing and research assistance

Lecture 5 Comms519 美国波士顿大学