When Metrics Turn Monsters ....

Darryl young, a 61 year old heart patient slipped in to coma after heart transplant procedure. His doctors continued to misinform his family regarding his chances for a year. All because his death before a one-year time window would have made the transplant unit’s number look bad

~Newark, USA [source]

“If you were a hospital administrator and you wanted to reduce the number of deaths in your hospital to make your hospital rise in the ranks, no question, you eliminate chest surgeries. And just by eliminating chest surgeries, you’re almost guaranteed to rise in the system as long as other hospitals are forced to do that chest surgery that you’re refusing, and that’s basically what happened in San Francisco”

~Dr. Michael Mann, commenting on systemic problems in VA(USA) [source]

If we use more time because we are nice and do the dishes or change a wound, or if the morning-care takes longer, we aren’t paid … Everything is based on economics

~A Nurse commenting on state of senior care in Norway [Source]

Vale of York Clinical Commissioning Groups (CCG) announced plans to delay planned operations for many smokers or patients with a BMI of over 30 by either 6 or 12 months if they could not prove they have stopped smoking for 2 months or lost 10% of their weight. More recently, CCGs in Hertfordshire announced an indefinite ban on surgery for smokers and obese patients

~Report By Royal college of Surgeons, UK [source]

While urgent cases are still being booked and seen by doctors, hospital staff are telling me less urgent categories are being delayed in the lead up to the end of the financial year. The result is fewer patients on outpatient clinic waiting lists and fewer patients moving through the system onto elective surgery waiting lists

~Allegations of gaming scheduling in Victoria, Australia [source]

“….We have a facility measure called >14 days from desired appointment. What it means is … .When we exceed the 14 day measure, the front office gets very upset, which doesn’t help us. …. You can still fix this and get off the bad boys list, by cancelling the visit (by clinic) and then re-scheduling it with a desired date within that 14 day window”

~An excerpt from VA employee email explaining how to game the system [source]

A few stories from bowels of healthcare system. Some amusing. Some downright tragic. But one can’t help but ask how could the noblest of profession let a patient suffer just for numbers ? How could the metrics meant to protect the interests of patients end up harming them ? 

Reality is certainly much more complicated than the usual romanticized notions of healthcare. To make sense of this reality, we need to understand why did we need metrics in healthcare.

From Babyloninan accountants to ratings of Uber driver, every industry in every era has relied on some kind of quantification to gauge performance.  And why not. Quantification introduces a sense of certainty to an otherwise uncertain world. With quantification one can measure, compare, and choose. Policy makers have a basis to plan, reward, and penalize. Without metrics, everything would be driven by anecdotes and intuition. 

Healthcare poses a unique mix of challenges. Why ? There is a tremendous information asymmetry and lack of transparency. One physicians judgement (read intuition) could be different than the other. Can one ever judge competency of a physician with reasonable accuracy ? One treatment doesn’t work on a similar patient. More often than not a patient has no way of knowing the exact financial liability of a medical treatment. Then there are these tragic cases of preventable of medical errors. Similar is the story of wasteful expenditure. Or for that matter healthcare’s decreasing productivity, and increasing cost.

If any industry ever needed a divine Saviour for salvation, it is without doubt Healthcare.

Metrics : The Genesis

And Saviour did arrive. It arrived in form of an idea riding a golden horse of data driven intelligence. Roughly captured it goes something like this. “Much of healthcare’s issues are caused by it’s uncertainties. The best antidote to uncertainity is of course certainity ! Which in turn comes from measurement. Lets start measuring. Then we can predict,judge, plan ….. ”

So the powers that be including bureaucrats, advocacy groups, experts set out to identify and define measures. The overall mission was to focus on patient experience, improve care quality all the while reducing cost -  the magical healthcare triple aim. Value became the mantra to live by. So much so that it spawned a new flavor of delivery systems – the value based healthcare, Accountable healthcare and so on. At core of this "new" system was simple mathematical expressions - the metrics.

Metrics : In Action

Metrics were created for Productivity, care process, and actual clinical outcome. As is the norm in healthcare, same problem is solved by different entities in different ways often creating new problems. Lets take example of messiest of healthcare systems – the US. Here overlapping metrics set are defined and maintained by multiple organizations including CMS, TJC (The Joint Comission), AHRQ, PCPI (Physician Consortium for Performance Improvement) and so on

Metrics began to be used to reward and penalize hospitals based on their performance . Performance of hospitals were compared and published by independent portals, directly impacting their ability to attract business. 

Metrics : The Impact

The first order impact of these metrics was more or less along expectations. If rankings are lower, the hospital would get lower elective business. If the 30 day re-admission rates are higher hospitals could attract penalties. If the 1 year transplant survivor rates are lower the entire transplant department could be on the chopping block . If the care deviated from a published guideline, the physician could get exposed to legal liability. 

But the impact had the potential to cause further ripples. Because it was not just revenue which was getting impacted, it was also the reputation and in some case the very existence and survival of a hospital was on line.

Metrics : The Reaction 

Systems are made up of Humans and Humans respond to incentives. Both positive and negative. Faced with such adversity, Systems and Humans did what they do best. Adapt and Thrive. Metrics transformed from means to reach a goal, to the goal itself. At the end of the day a metrics is just a mathematical expression. Which means both the numerator and denominator are targets to “influenced” to improve the final number

If hospital or surgeon ratings are linked to mortality, don’t operate riskier patients. If the wait times are getting too long, move the patient between departments. See patients in emergency departments instead of admitting them to game readmission rates. Stick to the published care guidelines even if your intuition points to the contrary. And the tricks to game the system goes on and on.

Metrics : The Consequences

Once such behavior get institutionalized and incentivized, everything reduces to a numerator or denominator. This is where the second order impact kicks in and we see the real negative impact.

• The culture becomes risk-averse. The riskiest case are nudged away from treatment options or at worst denied. 
• Risk averse culture leads to over diagnosis, over treatment and consequently more
• Managerialism takes over actual management and problem solving. Everything is a checkbox in a template
• Metrics become a way for policy makers to deflect responsibility 
• Fighting for patients comes pretty instinctively to doctors. Bureaucratic roadblocks enabled by metrics leads to frustration and subsequently burnout.

Why taming this monster so tricky ?

At the very basic level, the problem is we don’t have the core healthcare science figured down like laws of physics. We have made tremendous progress from days of Aristotle and Charak, but there is a still a lot we don’t know. Besides healthcare is not just about human body and biological lottery , but psychological health, social determinants, personal choices among others. 

Trying to dumb down such a system with all its complexities and unknowns in a few simple equations is bound to cause issues. Add to it incentivization, you have a real problem at hand. Jerome R. Ravetz a scientific philosopher,  In his 1971 seminal work Scientific Knowledge and Its Social Problems, says that “when the goals of a task are complex, sophisticated, or subtle, then crude systems of measurements can be played exactly by those persons possessing the skills to execute the tasks properly, who thus manage to achieve their own goals to the detriment of those assigned”. This later gave way to the more famous Goodharts law.

But all said and done, you wouldn’t want to live in a world where nothing is measured and nobody is accountable. It is too scary even a thought.

So what can be done ?

It is not that this problem has been tried to fixed before. Healthcare Metrics are now increasingly risk-adjusted. Though it cannot capture all of complexity, it alleviates some of it. But is cure for metrics – more and better designed metrics ? May be. 

But may be a better idea could be to walk in other direction and take a closer look at the metrics and incentive culture. How about we (de)incentivise only what is clearly an error or an issue. How about we capture all information make it transparent as much as possible, but use only a few metrics for administration ? How about we do away with concept of drawing arbitrary lines in sand like 30 day * rates or 1 year survival rates ?    What is the alternative ? you would ask. How about lowering the bar of quality to focus on the process to be followed and not on the actual outcome. 

The answers are not easy. But till then healthcare industry would be well advised to heed the maxim – Be careful of what you measure. You’ll get exactly that !

What do you think ?