Advice to the Premier (and journalists) on Understanding Data and Variation

Yesterday, I happened to catch a media Q&A session hosted by the Premier of Ontario on the province's COVID-19 response to-date. A reporter for the local news in Toronto asked if the Premier could explain why the latest figures show the number of tests had dropped to 10,000 on Sunday when it was over 17,000 the day before. Here's the Premier's response:

"Good question; I was on here last few days saying they're doing a great job, they were hitting 17,000 consistently, then all of the sudden you come in and you see the 10,000. I still say, you know, everyone is doing a great job, but when I look at the chart... it starts up here and goes all the way down, and there's certain medical officers in certain jurisdictions (keep in mind we have 34 medical officers across the province), some just aren't performing. I'm calling them out right now: You gotta pick up the pace. When you have a whole bunch of them, half of them, really exceeding expectations, and then you have some others aren't even putting the work in as far as I can see, so we need to hold these people accountable."

As I listened to this, a few questions of my own leaped to mind:

• Why did the reporter ask about just these two data points? What about the preceding dozens of days' worth of data? Are they even aware?
• Why did the Premier use the question as an opportunity to single out half of the medical officers of health as "exceeding expectations" and the other half for not even putting in the work?
• In any circumstance, wouldn't there always be some medical officers above and below the average?
• Ostensibly, these same medical officers of health participated in the previous days' test results: Were they slacking-off then, as well? If the results were say, 15,000 or 12,500, would they still be in trouble?

Of course, I know the answer and it lies simply in the way both the journalist and the Premier perceive the world and events in it as simple cause/effect machines. In this particular case, both were providing a contemporary demonstration of a common management dysfunction known as Management by Result (MBR) where individuals are held to account for the outcomes of a system they work within, forever reacting immediately on the last data-point. In his classic book, The New Economics, Dr. W.E. Deming explains how this leads to confusion and delay:

"The outcome of management by results is more trouble not less. What is wrong? Certainly we need good results, but management by results is not the way to get good results. It is action on an outcome, as if the outcome came from a special cause. It is important to work on the causes of the results- i.e. on the system. Costs are not causes: Costs come from causes."

The good doctor's Rx? Understand and improve the processes that contributed to the outcome by learning about the difference between common causes and special causes of variation so as to inform the type of action to take. Indeed, if there's one thing that anyone can do to markedly improve their ability to understand events in the world it would be to learn about the variation that occurs in any system and why not to overreact to each and every data point.

Understanding Variation in Data

In his 1993 book, Understanding Variation: The Key to Managing Chaos, Dr. Donald J. Wheeler relates a story about a conversation between his friend and colleague, David Chambers, and the president of a shoe company. The president had a chart on his office wall showing the "Daily Percentage of Defective Pairs", similar to the following:

When David enquired about the purpose of the chart, the president condescended that it was to know how his plant was doing - obviously. David then asked: "Tell me how you're doing." The president paused, thought, looked at the chart and replied: "Well, some days are better than others!"

What the shoe company president did not understand is why some days were better than others. Systems, like those in a plant or for COVID-19 testing, all present variation in their outcomes as consequence of how their components interact with each other. Some days will always be better than others: The role of top-management is to improve the system by managing swings in variation in their products and services to reduce the gaps that emerge. This is a fundamental tenet of quality: How well do the parts of our solutions fit together? How well do they work together? How well do we? The gaps we uncover are a by-product of variation.

I get the impression that the Premier and the hapless shoe company president have a lot in common, in this respect. I imagine, given his remarks above, he's shown an ever-growing graph each day that looks like this:

This is a poor visualization because it doesn't help us to ask interesting and important questions about the data beyond "Why did the last day's testing results go down, Mr. Premier?" The response will be, unsurprisingly, primed for blame. Moreover, it obscures whether the differences in the bars is routine or common-cause variation, or whether there are identifiable signals of exceptional, special-cause variation to investigate.

Interpreting Signals from Noise in Data

The alternative is, as I have written here previously, to present time-sequenced data as a process behaviour chart so it is easy for the user to separate signals from noise. I've been tracking the Province's testing data in this way since early March; here's what my chart looks like as of May 6th, 2020:

I can tell at a glance that this chart describes a system that's frequently beset by special causes of variation by each shift in the red process limit lines. Each shift or data point outside the red process limits suggests an opportunity for top-management to investigate with their teams: This is the strength of a process behaviour chart: Quick identification of places to investigate for special causes of problems and issues in the attendant systems.

I would suspect given the dominant style of management used just about everywhere that they can be attributed to decisions and interventions of top-management - what Dr. Deming would refer to as "tampering". However, for now let's just zoom in on the last twenty-six days' of data to put the reporter's question (and Premier's response) into better perspective:

I think even without understanding the rules how to interpret the above chart, it's fairly evident how irrelevant the journalist's question and the Premier's response are around comparing May 3rd's results to the previous high of 17,000 two days earlier: By removing the context of the entirety of the data, we cannot make a rational analysis. In turn, this encourages a binary view of the world where we'll forever pitch and yaw on each day's results, holding some medical officers as "exceeding expectations" while deriding others "not pulling their weight", instead of viewing them all as participants in a contiguous, whole system.

This said, the above chart also shows that there are far more interesting questions to be asked about what occurred in the Ministry and Public Health testing systems on the three occasions I've highlighted where exceptional variation was detected in Ministry data that caused the process limits to shift: These are the most economical places for the Public Health Unit and Ministry of Health leadership to begin looking for special causes to eliminate from the system before working on process and method improvements.

Advice: Learn About Variation, Use Process Behaviour Charts

Using the comparison of two data points out of context as the basis for making decisions is of little value if the aim is to improve. My advice to the Premier and journalists is simple: Learn about the relationship between systems and variation and how to illustrate it with process behaviour charts so you can see beyond the noise. Once you do, the quality of your questions and subsequent decisions about what you see will improve significantly.

Further Study:

Leaders from Montgomery County Fire & Rescue Service and Snohomish County Fire District detail how they’re monitoring the COVID-19 surge and preparing for the worst-case scenario (April 17, 2020)

Understanding Variation, Donald J. Wheeler, SPC Press, 1993.

Twenty Things You Need to Know, Donald J. Wheeler, SPC Press, 2009.

Measures of Success, Mark Graban, Constancy, Inc., 2018.