Tiny Numbers, Big Mistakes: A Scary Case of Medical Misinformation

Have you ever made a decision on limited information – a tidbit, a smidge of knowledge, or a gut feeling backed by just a little data?? We probably all have done this and sometimes either it was just necessary due to urgency or the decision was small with little consequence that it was ok to skip past all that fact-checking.? Fact-checking takes time, and when the data seems right, when it feels right why bother?? This is exactly what happened in a recent medical study and it’s a lesson for all about how data can be deceptive.

In June of 2020, the Agency for Healthcare Research and Quality (AHRQ) which is an official part of the Department of Health and Human Services started researching the impact of errors in hospital emergency department diagnoses.? They were curious about diagnostic accuracy and adverse events including death rates. It was a noble inquiry and many people do have a fear of hospitals causing harm, so why not take a look?

Small Becomes Big

When it came to assessing resulting deaths attributable to misdiagnosis it was a challenge.? Deaths are a very rare event and in fact, the study authors used only one 2010 study from a Canadian hospital that reported among 503 patients 1 death.? Just one.? And here is where the data dupe occurred. That sample of one death was drawn out using some basic math that just did not make sense.? The percentage (1 divided by 503 = 0.02%) was then multiplied by the total estimated Emergency Department patient visits in a given year, 130 million, resulting in a little more than 250,000 deaths.? The implication here is this represented the number of preventable deaths if they had just received better treatment.?

Could ER misdiagnoses really account for 7% of all deaths and be the 4th largest killer in America?

Quick Estimates Help

The math was suspect and the study’s data experts might have treated their small data samples with more caution.? The stunning outcome was the estimated number of deaths of 250,000.? A quick ‘gut-check’ or Fermi-like estimate might have benefited the authors and readers alike.? Take a reference to the total number of U.S. deaths annually of 3.4 million and that might have also given pause.? Could ER misdiagnoses really account for 7% of all deaths and be the 4th largest killer in America? More than accidental deaths, strokes, and respiratory disease? At what point does the unlikely become implausible?

Fears Drive New Cycle

The very same day the study was published, many of the major news organizations such as the New York Times published articles about the AHRQ report.? Eye-catching headlines about the number of ER deaths were frightening. They created that type of emotional response of fear that makes readers more alert.

The news cycle grew to include social media where the New York Times post was viewed 200,000 times. Fortunately, it was also quickly challenged by others who read the details of the AHRQ study.? However, at that point who knows how many readers and Twitter viewers took the headline at face value? ?Further, after all this time both the stories and Twitter posts remain online, unedited, uncorrected.? They still do today.

The AHRQ started with good intentions and months later is has been updated with comments about their methods. There was good design and research and there was even a peer review that notably pointed out the risk of making their conclusions.? They knew enough to be cautious with the data.? The data that seemed right, that felt right. The research team was data duped by small data samples, extrapolation, and their own confirmation bias.? In their bias, they only saw the data they wanted to see. In the media, the writers and editors only read the data they needed to see for a good headline. And for us readers? Only the fear. The fear of places where we normally seek safety and care. This is the power of data. This is how data can scare us, change our decisions, and maybe make us less safe than necessary.?