On the dangers of vanity metrics

More and more has been written on the distinction between vanity and clarity metrics in marketing. Vanity metrics involving big sounding numbers, often out of context, that imply success, but don't actually translate to any meaningful measure of effectiveness. Clarity figures are often the reverse, and frequently include contextualised ‘vanity metrics’ as well.

For many of us in the industry who are attempting to live on the ethical side of reporting on our campaign effectiveness, vanity metrics might be perceived as a harmless placebo for the less well informed. What harm can the vanity metrics do, right?

Well, let's look at some of the dangers.

Vanity metrics can lead to poor channel choices

Let's consider a fictional metric: we've got 30,000 Facebook impressions for ￡150 of media spend. Sounds great right? Well, we may not know if it's good, even within the platform. What should ￡150 of media spend buy you on Facebook, for a given audience? How does it vary depending on targeting, competition/bid rates, etc?  Do we know if the targeting was good? 30,000 people outside core target segments might not be terribly useful for this form of promotion. How does it compare with another creative that you were A/B testing?

So vanity metrics here can be misleading. Note that if you add context (the audience was right, the relative efficiency of spend/impression was good vs other channels, the creative had been tested and was shown to be effective / drive reach) – the vanity metric becomes a clarity metric.

Vanity metrics can lead you to poor creative choices

We once started working with a brand where we were asked to launch phase two of a campaign which had been devised elsewhere. We were at something of a loss as to how it would land; it was not the strongest story. So we asked for a campaign report for phase one. And were given a document highlighting 24 pieces of coverage (which were in fact BusinessWire syndications) with a supposed reach of 726 million!

It was clearly nonsense. Yet, if anyone had believed the success of phase one on the basis of these metrics, it might explain why the campaign had a very similar phase two. And explain why it was being rolled out to the same channel for a second time. The definition of madness being doing the same thing and expecting a different outcome.

Vanity metrics can lead you to expect unrealistic outcomes

Just as you might expect outcomes from exposure in publications with supposedly vast readerships, influencer marketing can also represent another challenge for vanity metrics. An influencer might have hundreds of thousands of followers. And yet, anyone with a Twitter account can tell you, this isn’t always reflective of their influence. A substantial proportion of social media users are inactive, or at best, such light users that the odds of them seeing a post from your influencer are vanishingly small. So if you believe follower numbers, you might expect an avalanche of sales or enquiries following an influencer repost... And you might be disappointed. True reach – deduping fake followers, inactive accounts and bots – is a much more meaningful indicator, but even then there are other metrics that matter when assessing influence.

These metrics can also confuse the purpose of the activity, which might not necessarily be about reach in raw terms, but instead intended to drive other outcomes. For example, you might want to assess the impact of the activity on changing brand perception, beliefs or behaviours in a target audience, things which need to be measured differently.

Vanity metrics can underperform in cases of real value

Data absolutely can and does lie. Consider a digital marketer making the case for more budget. They might observe that the volume of web traffic directly referred from earned media is relatively low. Compared to PPC, or any number of other channels. Certainly when considered relative to the level of investment. So should digital get all the spend, as Gary Vee suggests?

Of course this is completely misleading. The primary goal of most earned media coverage is generally brand exposure, not direct engagement. The value you get comes from other effectiveness measures - brand awareness, perception etc amongst relevant target audiences – which need to actually be measured year on year if possible. Traffic is cherry on the cake, not the defining factor. So vanity metrics can be dangerous here.

Vanity metrics favour short vs. long, efficiency vs. effectiveness

The plethora of metrics available from digital platforms in particular, but also potentially misleading media measures like AVE and "circulation" or "monthly unique users" often send us down a path to efficiency. In what way can we spend the least money for the highest (tactical) numbers?

Note that these numbers might indicate some degree of performance, but they do not describe marketing effectiveness. If you're not gaining or holding market share, your marketing is not working (regardless of how 'efficient' your spend might be). Good marketing has channel and creative choices that are reflective of a coherent strategic approach, addressing the problems your brand is facing in the market. Don't get pulled in by the glossy sheen of the 'big number' vanity metrics; instead look to the more commercial, more tangible measures of success and hold those up. Effectiveness first, then efficiency.

In some cases, vanity metrics CAN encourage further experimentation. It can justify further investment in channels and tactics that are still new and untested for many organisations. But they can also lead to a rapid death if the wrong metrics are specified as objectives or measured as outcomes. It's vital that the people governing strategic decisions about marketing investment have the clarity metrics to hand.

I'd love to hear other thoughts of where vanity metrics have helped or hindered, and what people are doing to help shift the thinking away from the nonsense and into meaningful effectiveness measurement. If you're interested in this line of thinking, I continue to heartily recommend 'How not to plan' by Les Binet and Sarah Carter. It has an excellent section on data, and on media spend, chock full of examples of where and how data can be used - and abused. And this post via LinkedIn is quite good on how to make use of vanity metrics too.