Measuring the Wrong Things

It is interesting that in America they order the Olympic medals table by number of medals whereas in Europe we order it by gold medals. In the end it made no difference to who came top for the Paris Olympics (the USA would be top on either measure – with any reasonable tie-break), but it certainly has an effect further down the table. In the American version for example, the UK came third, but it only came seventh in the European version. Is this difference really important? Perhaps not, but perhaps it is if it is a factor in determining which sportsmen and women receive funding.

This made me think about who we should say is the best Olympian ever. Perhaps we should look at the total medals won; in which case the clear winner would be Michael Phelps USA (2004-2016) winner of no less than 28 Olympic medals (23 of which are gold). This is some distance from the next best, Larisa Latynina a Soviet gymnast (1956-1964) with 18 medals (9 golds). But does this miss something? Are we measuring the right thing here? Phelps’ achievements are amazing, but when I think of what epitomizes of the Olympic games, the 4x100m medley in the swimming isn’t the first thing that springs to mind.

This problem reaches into other areas of life. With the increasing collection and analysis of data, metrics are everywhere. Whether in academia, business, or public policy, we're constantly measuring things to track performance, set goals, and make decisions. But what if the very act of measuring is leading us astray? This is where Goodhart's Law comes into play, famously stating, "When a measure becomes a target, it ceases to be a good measure."

The Academic World: When Citations Go Wild

Let’s start with academia. Universities and researchers are under immense pressure to publish papers and get cited. Why? Because metrics like citation counts, h-indices, and impact factors are often used to measure academic success. But here’s the catch: when these metrics become the primary focus, they start to distort behaviour.

For instance, researchers might start choosing trendy topics that are more likely to get published in high-impact journals rather than pursuing truly innovative or risky ideas. There’s also the issue of “citation padding,” where authors cite their own work (or their friends’ work) excessively to boost their citation counts. The result? A flood of papers that might look impressive but don't actually contribute much to advancing knowledge.

Goodhart's Law: Examples in Business

Goodhart’s Law isn’t just an academic issue. Take sales targets, for example. Many companies set specific goals for their sales teams, like selling a certain number of units or reaching a revenue milestone. On the surface, this seems like a great way to motivate employees and drive growth. But when the target becomes the sole focus, things can go wrong.

Imagine a sales team under pressure to hit a quarterly target. They might start pushing products onto customers who don’t really need them, offering heavy discounts that eat into profit margins, or even manipulating sales data to make the numbers look better. These tactics might help them hit the target in the short term, but they can harm customer relationships, brand reputation, and long-term profitability.

Another business example is in customer service. Companies often measure success with metrics like average call handling time or the number of cases closed. But when employees are judged strictly by these numbers, they might rush through customer interactions or close cases prematurely, leaving customers dissatisfied. The metric is achieved, but the actual goal—providing excellent customer service—is missed entirely.

Public Policy: When Metrics Miss the Mark

Goodhart's Law can also cause problems in the public sector. Consider healthcare, where hospitals might be judged by metrics like patient throughput or the average length of stay. In the UK, for example, hospitals have a target of patients being seen by a doctor within four hours. In practice, this often means patients briefly see a doctor just before the four-hour deadline and then wait for much longer before they receive full treatment. The metrics might look good, but it isn’t necessarily leading to better patient care.

And think about education, where standardized test scores are often used as a measure of school performance. When these scores become the primary target, schools will tend to "teach to the test," narrowing the curriculum and neglecting broader educational goals like critical thinking and creativity. Students might end up with high test scores but without the skills they actually need for the future.

Finding a Balance

So, how do we avoid falling into the Goodhart's Law trap? The key is to use metrics wisely, understanding that they’re tools, not goals in themselves.

1. Use Multiple Metrics: Relying on a single measure is risky. Instead, use a range of metrics that together provide a more complete picture.
2. Focus on Outcomes, Not Just Outputs: Metrics should reflect the ultimate goals you’re trying to achieve, not just the steps along the way.
3. Adapt and Refine Metrics Over Time: What works today might not work tomorrow. Be ready to adjust your metrics as circumstances change, and monitor for any unintended consequences they might be causing.
4. Encourage Ethical Behaviour: Finally, promote a culture where doing the right thing is as important as hitting the target. This is far from easy when performance assessment is mainly based on a set of metrics.

So, metrics can be a useful tool, but let’s not assume that putting them in place means we are being objective. We need to take the time to look carefully at what the metric is telling us, and more importantly, what it isn’t.

Goodhart’s Law is a powerful reminder that metrics are a double-edged sword. When we turn them into targets, we risk losing sight of the bigger picture. Whether in academia, business, public policy, or Olympic funding, it’s crucial to use metrics as guides, not as ends in themselves.

Returning to the subject of the best-ever Olympian, I find it difficult to look any further than Usain Bolt. You may ask how that can be when the medal count for Phelps is so much higher? First, let’s consider the opportunity for medals. 16 out of Phelps’ 23 Olympic medals were in medleys, relays or medley-relays. The only relay available to Bolt was the 4x100m, and there’s no opportunity for medleys in athletics. Second, only one of Phelps’ 23 gold medals was in an individual freestyle event (200m freestyle, 2008) – i.e. in an event which determines the fastest person over a particular distance. There’s only one style of running (or perhaps two if you include hurdles). Finally, there’s something quintessentially Olympian about the being the fastest person on the planet, and there was no doubt in anyone’s mind who that was between 2008 and 2016. It was just a question of how much Usain Bolt would win by, wasn’t it?

As an example of Bolt's draw, on the day of one of the 100m finals I was driving back from a short break and broke my journey in Cambridge, having to find a parking space and then walk to the main department store, so I could watch the race on TV in the electrical goods department. I was not the only one. Quite a crowd had gathered to watch Bolt. It quickly disbursed after the race. If they had been using a metric of sales per person visiting the department (not an unreasonable metric for department store sales staff), they would have scored poorly that day.

References

• Macleod, M., and Wykes, T. (2023). "Goodhart's Law in Academic Research: How Metrics are Distorting Science." Journal of Research Metrics.
• Savic, G., and Celikovic, M. (2023). "The Role of Performance Indicators in Higher Education and the Impact of Goodhart's Law." IPSI Transactions on Internet Research
• ResearchHub Foundation. (2023). "Overcoming Goodhart’s Law in Scientific Research."