Three Graphs You'll Never See Used to Show Ontario's COVID-19 Response Data

What can you learn from three charts that an abundance of reports and media "analysis" on the last data point or two in a series cannot? My aim with this short post is to show how visualization of a system's data in-context and over time can generate more insights than are typically offered by leadership or the media. I'll be using three time-series charts that I've made using Ontario's #COVID19 Response data to make the case: The first two are Process Behaviour Charts that describe when test samples are taken and then are processed; the last shows an overlay of the two resultant curves. All data was taken from the Status of COVID-19 Cases in Ontario database.

A brief overview to the uninitiated: Process Behaviour Charts are used not only to visualize discrete data points, but to provide a means for separating signals from background noise among them, chiefly process limits that are calculated from the dispersion of the data points themselves (traditionally shown in red). In each chart, when you see the solid red process limit lines move in stair-step fashion up or down, something has changed in the underlying processes that is out of the ordinary for that period of time. More shifts in the limits means more unpredictability and instability in the system.

This first Process Behaviour Chart shows COVID-19 Test Samples taken over time, from March 5th to the present, with three daily target lines corresponding to the figures that are often mentioned in the media interchangeably: 16k/d, 20k/d, and the top-end theoretical maximum of 25k/d.

What can we learn from this?

• The system is unstable and unpredictable - at least, over the long-term. We know this because the process limits have shifted seven times. This tells us there have been changes in capability and processes. Unsurprising when you learn that this is a system which has been built mid-flight.
• There are two distinct periods the system operated, separated around April 15th, which happens to coincide with when the Province changed how it reports tests, from one person one test to total test samples taken.
• Whereas the earlier period seemed to fluctuate randomly, the latter period follows a pattern of rolling hills, with each nadir corresponding to a Sunday or Monday and each zenith on or about a Friday. The regularity suggests that we may be seeing more about how tests are reported than how they are administered.
• The gap between the process limits in the last three periods is substantial, which tells us where we can expect to find 99%+ of the data fluctuating. If your aim is to have a system achieve a standard as consistently as possible, eg. the daily targets, you want to improve your processes to control variation around that target. What we see here is a system that achieves the targets as an exception to the rule, perhaps at some other cost we can't see right now.
• For the period of April 30th to May 17th, the system's variation fluctuated around a mean just below the 16k/d target, and was close to hitting the 20k/d target once on May 8th. However, this was a temporary condition: System data tends not to obey arbitrary targets.
• For the last period of May 27th to June 9th, the system's variation moved to a higher mean of ~18k tests/d, and exceeded the next target of 20k tests/d four times. Again, not a regular occurrence because the systems variation doesn't permit it.

The next Process Behaviour Chart uses the same time frame and tells us about what happens after test samples are gathered and in-process:

What can we learn from this?

• As with the prior chart, this one describes a system that is unstable and unpredictable, featuring nine shifts in the process limits.
• Two different curves are shown: One that looks like a very non-random distribution and the other moving up and down in a semi-regular fashion. These correspond to when the Province relied on their central labs, and when they opened up testing to independent and hospital labs.
• Both curves show active management of the underlying processes that regulate the queues and how they are reported. The nadir for the later period curves tend to fall on Sundays and Mondays and zeniths on Thursdays or Fridays, closely mirroring what's happening in the Test Samples PBC. This reveals the two systems are closely linked and managed.
• The gaps in the variation for the later periods are larger than those for testing system, which tells us that the queues expand and contract at rates dependent on the internal processes. This isn't surprising when you consider the different types of equipment used and the rates they can process test samples.

When I was compiling the two charts, the mirroring between them piqued my interest enough to want to overlay the two curves to see the degree of sympathy between them:

Note the initial discrepancy in the two curves on the left side from March 17th to 30th and then how they begin to move in sympathy with each other, with the backlog queues slightly trailing the test samples. This isn't random or incidental: They are being actively coordinated as well, which is remarkable given the size of the system which encompasses thirty-two Public Health Units and hundreds of independent and hospital labs! What does this suggest about how the system is being managed?

Visualizing data over time and in-context, along with some theory of why the figures fluctuate (short answer: all systems have variation) take us from being passive consumers of figures to active learners, even when we're not able to directly see into the system's inner workings. As shown above, we can learn a lot by plotting data in a Process Behaviour Chart to aid our interpretation, which in turn can suggest other charts to illuminate the dark corners of our understanding.

Further Reading for the Curious

Understanding Variation, Donald J. Wheeler, SPC Press, 1993.

Twenty Things You Need to Know, Donald J. Wheeler, SPC Press, 2009.

Measures of Success, Mark Graban, Constancy, Inc., 2018.