The Perils of Over-Optimization: A Parable

I’ve been telling this story for years in some form or fashion, and it never ceases to bring a smile to my face…but I'm also concerned that its lesson is still relevant. It was most poignant for marketers only running media channels and tactics that deliver a last-click ROAS without any consideration of incrementality. But in today’s landscape, with Google and Meta pushing nothing but ‘broad targeting’, ‘consolidation’, and bundled products, I think some marketers are jumping at these new ‘best practices’ as those of yesteryear (realistically like 5 years ago) did with branded search, running on seven different retargeting platforms, and disdain for anything resembling mid- or upper-funnel media.

That said, I’m pleased to see that changes in privacy law, consumer sentiment around their data, and what brands like Apple are doing, are bringing some thoughtfulness back to digital marketing, even if there is short-term confusion in defending marketing’s impact. Gone are the days of performance marketers hanging their hats on Google Analytics reports to guide budget conversations with finance teams. And that's a good thing.

For those marketing leaders wondering why actual business revenue is declining while performance marketing ROAS looks great? This story is for you.?

Meet Bob and Johnny

Once upon a time there was a successful toy store that had been a staple in its city for generations. As the owner, Bob, was approaching his 70s, he struggled with the decision to retire or stick it out a few more years. But one day, a young man approached him at the store wide eyed. He wore pants that sort of looked like sweatpants but also a button-down shirt, and glasses that Bob thought looked like they didn’t even have any lenses in them. When Bob asked if he could help the young man, he responded “I’d like to buy your store. I’m willing to make you a great offer.”

Johnny, the young man, had done his research. He saw the success Bob had built over the years. The store was always full, people recommended the store to all their friends, and the customer base was extremely loyal. Bob knew his customers by name and could adapt the shelves to keep up with changing customer preferences. Johnny was impressed but was even more excited by what he could bring to the business with his expertise in performance marketing. Bob had built a good foundation, but with some optimization, Johnny could take this thing to the moon!?

Bob decided to sell and set sail on a retirement cruise. Johnny got to work. At first, he just observed. There was some skepticism about new ownership, but business really was great. But, Johnny noticed some immediate red flags and began building a testing roadmap.?

Customer Data

Bob had notebook after notebook of customer data, including rich qualitative research. Silly, Johnny thought, there’s nothing in here I can optimize. I need to start collecting useful data so I can run tests and see how my metrics improve.?

Sales & Direct Mail

Johnny noticed that any time Bob had run a sale, it brought a wave of customers into the store immediately. Sales would be key, thought Johnny. Bob had always used direct mail to advertise his sales. But there were two things wrong with that in Johnny’s mind: Bob only sent mailers to people not in his CRM database. How crazy! It was current customers who were easiest to get into the store to buy, not people who had never even heard of the store. This was clearly in the data. Second, the people that Bob was mailing to lived all over the city. It could take them a while to decide to come into the store, so what if he could activate a more ripe audience? In fact, Johnny could see that those who lived the closest actually were the best buyers during sales. So Johnny found an opportunity for simple optimization: He would shift from mailers to flyers he could hand out to customers right outside the store walking by! Those are the customers who could take advantage of the sale right away.

The next week, Johnny put his test into market, and it was a huge success! Bob used to send out 3,000 mailers each time they had a sale and only 300 new customers would come in and buy. But Johnny was able to hand out only 1,000 flyers and 500 customers bought! This was a huge leap forward for the business: Conversion rate immediately went from 10% to 50% with one optimization! Johnny thought of Bob doing things wrong his whole career. “Old school marketers,” he laughed.?

Johnny knew the risk was making this a one-time optimization so he vowed to make this the new evergreen strategy. He put up a huge sign right in front of the store that said “Big Sale. Today Only. Buy Today! 50% off.” Under the sign was a large stack of coupons. This would get all walkers-by into the store to take advantage of the sale. While he watched the take rate of the sale stay way above Bob’s old metrics, he shifted his views to the shelves.?

A/B Testing

Bob had completely neglected the number one rule of performance marketing. Always. Be. Testing (with A/B Tests!). So Johnny decided on an age-old test he couldn’t believe Bob hadn’t run: red shelves vs blue shelves. So simple! That night when the store closed Johnny got to work. He painted half the shelves red and half the shelves blue. He sort of knew that blue shelves would work best since he knew a bit about the psychological impact of colors (not his first rodeo) so he naturally put some of the best-selling products on the blue shelves. By opening the next morning, the test was ready. While customers were a bit confused by the sudden change that stood out like a sore thumb, they didn’t really mind - after all, they already knew what they wanted. But Johnny was excited after a week to do the analysis. He collected all the data, organized it, and pulled up an A/B test calculator (thanks, Neil). He held his breath as he inputted the data. The results were in: 56% of people chose products off the blue shelves! And the data was statistically significant! Unbelievable! From now on, blue shelves only. It felt great to get a key learning under his belt.?

Panic sets in. Cut anything not clearly driving revenue.?

With these few changes, Johnny decided to sit back and observe a bit more. He knew he needed a cool-down period after a couple of these big changes. He was immediately a bit concerned though. What used to be a steady incline in revenue was now barreling down and taking his profit with it. This seemed strange: Every decision he made had been data-driven and had measurable efficiency gains. After a few more months of observation, Johnny started to get a bit worried and had a brainstorming session to turn things around. He needed to be more strategic and think like the owner he was.?

A quick look at the rest of his expenses and he had it! Earlier he had overlooked the billboards, local TV, and community sponsorships Bob had been running all year long. Billboards? Johnny had the big sign out front when people were actually close enough to come in. Event sponsorship? Who’s in the mood to buy toys when they’re running a charity 5k? TV? Ha, what is this, 1999, he thought? Cut, cut, and cut.?

Next was staff. He had salespeople on the floor who helped customers find what they were looking for and were beloved by customers for their knowledge and expertise. Bob had noted this in his notebooks, but Johnny ran the numbers and realized that only the cashiers actually made the store any money. Why have employees that didn’t make the store any money? It was SO measurable he couldn’t believe he hadn’t seen it before! Salespeople were retrained to be cashiers so they could make the store more money! Phew, thought, Johnny, glad I caught that.?

Customers seemed lost though. Existing customers came to the registers upset. “I can’t find what product I want and I can’t get anyone to help me!” they’d yell. New customers seemed confused, and unclear as to why they even came into the store in the first place. “Some guy grabbed me out on the street, threw me, and landed me here!”?

As they’d start leaving the store angrily the cashier would yell, “Wait, don’t go! Here’s a ‘buy one get one free’ coupon that you’ve triggered by leaving the store.”?

The End.