The problem with chasing numbers...

What can COVID-19 teach us about Performance Management in Business?

As we watch the Worldometer numbers each day we are watching some of the most important KPIs ever in our history. What is evident is how quickly KPIs of all kinds can become a game of chasing numbers instead of being a means to understand and improve what we do. And that has never been so important as it is now. How does that happen?

The reporting and actions of governments through this pandemic illustrate just how difficult it is to make the best use of KPIs in performance management. The Worldometer metrics highlight many of the practical issues in choosing KPIs and flags up some of the basic data issues that lead to poor outcomes for strategy and decision-making:

• Are you solving the right problem i.e. have you properly defined the objective or outcome?
• Is the measure the best or most appropriate milestone in leading to that outcome?
• Is what you actually measure an accurate reflection of what you are trying to measure?
• Can you use the measure in time to make required decisions and changes along the way or does it only provide a “rear view mirror” view of success?
• Do you have an objective calibration of “good” or “poor” performance using the measure?
• Can the measure be gamed by the people/system being measured?

What are the objectives? Are you solving the right problem?

Ideally you want to measure the things that directly affect the objectives or outcomes you are looking for. But we see that what appears to be a straightforward enough objective becomes very complicated as soon as we try to define it in a way that can be measured. These were two of the UK government's stated objectives:

Save lives. What does that mean? Save lives today? Save lives tomorrow? Save all lives? Save the lives of people with COVID-19?

Clearly in a pandemic the goal is more critical, but the business equivalent is not so different from saying something like “increase marketing ROI” or “meet sales targets”. That doesn’t even begin to address things like the quality of lives saved or the number of people who stay alive but need critical care. It doesn’t address the issue of intensive care needs or the number who get very ill vs. the number who actually die.

Protect the NHS. Who or what are you protecting? The people in the health system? Workers or patients? Or are you protecting the system from the patients? It is easy to say that you don’t want the health system to be overwhelmed but that can easily (and has) resulted in people not going to hospitals who should, people only being treated once they are already critically ill and many other forms of illness simply being ignored through lack of focus. It also led to the disastrous decision to move  people back into care homes without testing, simply shifting the deaths elsewhere.

In the case of businesses outcomes are fortunately seldom fatal, but the issue is not dissimilar to managing call centres to minimise call times or to enact lean process management to make operations easier as opposed to managing for customer outcomes. It is easy to decide that a single milestone is the direct cause of the ultimate objective, but it is also easy to focus on the target instead of what the target is supposed to represent. It is worth remembering that “there are no side-effects, only effects”.

Reducing objectives to simple "cause-and-effect" targets is enticing, but that often excludes more nuanced and detailed approaches that address the entire system and take account of the broader consequences of different actions.

Which KPIs should we use to measure success and can we measure accurately?

Worldometer uses core statistics on “confirmed COVID-19 cases” and “COVID-19-related deaths” as the base of their reporting. Once again, what does that mean exactly?

Confirmed COVID-19 Cases. The number of cases identified and confirmed is a proxy for the actual number of cases in the country and is hugely dependent of how many tests are carried out, where and on whom. As that changes over time trends in positive tests become a poor measure of how many more or fewer cases there actually are in the population.

• “Confirmed cases” do not represent all cases. Unfortunately, the number of cases identified is a sub-set of total cases depending on how much testing is done so the number is always low but we don’t know by how much. They are also not a representative sub-set of all cases so cannot be used to draw any conclusions about cases in the population at large. Comparisons also become impossible. In Germany more young people have been tested. In South Africa only a small number of tests have been carried out at all.
• The number of tests changes over time, but not in a consistent way. If the number of tests is increasing (and then fluctuating) on a daily basis there is a link between increasing number of confirmed cases and the number of tests, but not a known or consistent link. That makes looking at trends irrelevant in a way since there is no correction for a “base-line number of tests”.
• Who is being tested and where changes over time. If tests are kept to frontline workers then we can expect to see a higher number of cases per 100 tests than if tests are random within a population. Similarly, if only those with severe symptoms were tested initially but over time a wider set of symptoms lead to testing, then a simple re-basing is not possible. Ditto for gender, age and other personal characteristics.

COVID-19 Deaths

• How do you define a COVID-19 death? There are deaths with COVID-19 and deaths from COVID-19. There are deaths in hospitals or elsewhere. There are deaths from people who have received a test and some where no test was done but symptoms were recorded. The way deaths are recorded, and the way causes of death are recorded differs by country so once again trends and comparisons are meaningless.
• What about “excess deaths” as a measure? In some cases it is argued that many of the people dying have co-morbidities so that COVID-19 is bringing forward a death likely to have happened in the near future. In that case the disease has resulted in a premature death but over the full year of deaths the number would be the same. In other cases, such as frontline staff or younger victims the reason for death may be from exposure to a high viral load or related to specific underlying conditions. In the longer run, a final view may include deaths that result from the change of focus (away from cancer and other illnesses) and the effects of interventions taken.
• Deaths in “the last 24 hours”.  If you want to know if deaths are declining then you want to know how many people actually died in the last 24 hours. The reality in the UK is “deaths we have found out about in the last 24 hours”. Statistics are still being taken using the formal procedures for registering deaths. That can take up to 2 weeks or even longer. People can be ill for a long time before they die and then the death can take anything up to three weeks to appear in the statistics. That lag has huge implications for the understanding the link between the actions taken (e.g. lockdown) and the outcomes we say are needed before making changes.

Business KPIs suffer from similar issues. The measures we have for digital marketing campaigns are immediate but focus on very short-term actions, whereas we know that measures of real sales increase or improvements in brand equity take time to show up depending on purchase cycles and the power of the campaign. Even where measurement parameters are carefully defined, it is often impossible to identify “objective truth”.  

What is “good”, and can you game the system to give you the answer you want?

Once you have agreed all the measures, how do you then define what constitutes “good”? We already see that opinions of how well different governments are doing in managing the crisis has no objective answer. Perceptions vary depending on prior beliefs (political, economic and social) and there is no way to know what the alternative reality might have been (how many lives might have been lost if….). Even if “good” is defined in advance (in the UK “we will have done well if deaths are below 20,000” vs. “there will be 500,000 deaths if we don’t do something”) the list of extenuating circumstances and issues beyond one’s control is long and even longer when the results are not favourable.

Again, if we compare with businesses, there is a tendency to reposition the measures of “success” depending on the outcome. Witness Tesco removing Ocado from the competitor comparison retrospectively in order to meet their bonus targets.

This is a typical issue in performance management. People all filter information and choices in their own best interests. This may be for unconscious reasons, rather than a deliberate attempt to obfuscate, but it happens in all performance management systems. Ultimately rewarding people or holding them to account based on pre-agreed measures and objectives is not nearly so straightforward as it seems.

So why have these measures at all?

These or other KPIs are useful because, in spite of their shortcomings, thinking about what is important, setting a goal and assessing how well you are doing in achieving it is important in making progress. The problems arise when we forget that they are only indicators or proxies in a large and complex ecosystem. When we start to “believe our own marketing” and focus only on chasing the numbers we are in danger of creating unintended consequences that lead away from, rather than towards, the outcomes we want.