Of Cobras and KPIs

During the British rule in India, the city of Delhi was once facing a problem of venomous cobras. In an attempt to solve this, the governing authorities offered a bounty on cobra skins - get a cobra skin, get money. Here is what followed -?

• It began well with locals handing in cobra skins and getting rewards.
• Soon, the amount of cobra skins coming in had increased several times.
• It turns out that people had started farming cobras just for the bounty.
• Authorities realized this and scrapped the scheme.
• Due to lack of a market, the farmers released the cobras into the wild.
• The cobra menace, instead of getting solved, worsened significantly.?

Another story, much closer, from LinkedIn

LinkedIn introduced endorsements around 2012 - a product that touches upon social as well as hiring aspects of the platform. Endorsements are used as one input to rank search results for skill experts, so that recruiters can find candidates with the right skills. The North Star metric was ‘Total Endorsements’. To improve this metric, LinkedIn designed interventions that allowed users to very easily endorse another user. Soon, the volume of endorsements increased significantly. By 2017, there were more than 10 billion endorsements with more than 43 million endorsements given each week. This was not a resounding success, though. The value of endorsements diluted and recruiters could not trust these endorsements as a measure of expertise.

The Cobra Effect and 'gaming' of KPIs

There are several more examples of what is called the ‘Cobra Effect’ - referring to unintended consequences of your success metric. Choosing KPIs and especially the North Star is never easy. This is indeed one of the most important choices we must make - we have heard, 'you can't improve what you can't measure' (Peter Drucker) and also, 'you are what you measure' (Dan Ariely; HBR 2010). There are many considerations, and the topic is too involved for a single article. An additional question that you need to ask is - can the KPI be 'gamed'?

'Gaming' doesn't necessarily have to done be by the Product Manager who can influence it to make the product appear successful. It could very well be the behaviours of the end customers that don't serve your purpose. It could even be bugs in the system that improve certain metrics while degrading the overall customer experience.

At Bing, the Microsoft owned search engine, an important KPI was ‘queries per user’. Once, out of the blue, without any improvement/intervention in the experience, the queries per user shot up. Big success? Shortly after, a bug surfaced that had degraded the search relevance. It didn’t take long to realize that due to poor relevance the users ended up making more queries and 'improving' the KPI for a degraded customer experience. The team reconsidered the metric and switched to ‘sessions per user’ instead.

Towards better performance tracking

Is there a KPI that can never be gamed? Unlikely. Part of the solution is in choosing better KPIs that are less likely to be gamed. Another part is in having the right set of supporting metrics and reviewing KPIs.

• To begin, when you and your team are brainstorming over the right KPIs to track, prefer KPIs that measure 'quality'. KPIs that measure volume/count aren't often great and are more prone to gaming. KPIs measuring quality are often ratios.
• Choose KPIs that are closest to the intended behaviour / 'perfect experience' of the user. The KPIs should take a hit when the experience degrades. YouTube chose watching minutes as its North Star. Many online retail platforms choose Customer Lifetime Value (CLV).
• Keep reviewing KPIs and ensure they are aligned with business objectives. Objectives keep changing at different stages of maturity of an organization/product. Indeed, it is common that many products/organizations initially have a customer acquisition objective and later focus on retention.
• Segment your KPIs. De-aggregate your views and don't miss out on pockets where the performance is cause for concern. Alerts for performance drops for all segmentations may be overkill, but can help in the case of some key segment views.
• Have strong guardrail metrics. Some guardrail metrics should be those sacrosanct metrics that must never be degraded. Others could be metrics that understand and capture the dependence between key metrics.
• Track system performance metrics and enable alerts for bugs.
• Track input metrics and driver metrics. Not all of these may be in your control and aren't therefore actionable. But tracking helps quickly isolate issues. The best input metrics are those you can influence and are good leading indicators of the output.

The points above don't guarantee safety from the 'Cobra effect', but adhering to these may significantly reduce your chances of falling prey to it. Additionally, they should give you a 'reliable' view of performance that you can base decisions on.

If you are looking for a good, quick read on KPIs in general, this Forbes article by Jacob Drucker is a good starting point. Read some other interesting examples of Cobra effect here.

Got any interesting examples of the Cobra effect that you came across?