Models + experiments for Strategic Analytics

Last week saw a wonderful workshop by the Marketing Science Institute: 'Comparing the Value of Field Experiments and Statistical Models for Strategic Analytics.' The questions and panelists were handpicked to get an engaging discussion about marketing attribution:

1.???? When you think of an experiment, what does this mean to you?

2.???? When do experiments work well for you? What are your constraints?

3.???? What do you do when an experiment is not “statistically significant”?

4.???? When are you using econometrics and models to measure marketing effects?

5.??????How difficult are these methods compared to experiments?

6.???????What are some use cases for these methods? How does MMM fit in your portfolio

7.????????How do we build support across our organizations when it comes to actioning?

?Among the panelists, 2 are renowned for their marketing mix modeling, 2 for their experiments (randomized controlled trials), and 2 for their practical implications. Fireworks in the forecast!

?

Interestingly, a broad consensus emerged:

?

1 Start from the use case: what is the business problem you are answering? ‘It’s not just RCT or nothing’ and ‘All RCTs are wrong, some are useful’. For instance, many advertisers asked Amazon for an RCT on a specific campaign, and when it turned out to give insignificant gain, they would stop advertising with that ad product, or even on Amazon completely. If that was the business decision, other tests should be run.

2. Statistical models are fantastic to identify which areas are most important to dig further into (eg with a field experiment): in which channels and locations are you historically underperforming? For instance, in my Practice prize paper, we found that direct mail represented about 80% of the company’s marketing communication budget but had negative ROI. We investigated why and found that the content resonated, but the company had increased spend way past diminishing returns. Cutting that expense in half brought it back to profitability.

3. Marketing is rarely a life-or-death question. When the experiment is not statistically significant, it can still point to the better decision. Rick Bruner, the CEO of Central Control, had a wonderful slide to this point:

?

4 Statistic models are difficult to get right, at least as difficult as correctly running experiments. As Rick Bruner said ‘stop saying experiments are hard . So is MMM. And so is MTA’

?5.???? Organizational support is key: from Perno Ricard to Pinterest and Altria, panelists discussed the long road to buy-in for specific methods

6.???? Frequent client mistakes in MMM are demanding too much granularity. Insisting on different estimates for each campaign brings multicollinearity.

7.???? Geo experiments have plenty of benefits, including not needing personal individual identifiers and employing almost all media, similar to MMM. With a stacked design, as implemented in Central Control's Rolling Thunder, you are able to stop in a location where sales effects are negative.

?

The case study of Central Control involved a top 20 US advertiser who gets about 20% of their $55B revenues from online sales, 80% through phone calls (similar to Inofec in the Practice Prize). They spent $100M a year on branded search and suspected almost none of the resulting sales were incremental. Therefore, the CMO wanted to cut that in half.

The geo experiment revealed that sales actually went up in the online channel when branded search was turned off!

But the company saw a big drop in sales from phone calls. Turns out that the branded search ads had the phone number above the fold, making it easy for people to remember when they were ready to call. To save $50M in ads, the company put $1.5B at risk in sales.

That's a wrap for me! I took a nice walk to Hudson Yards to meet up with current Amazoniands: