The ISO GUM Has Always Been Biased In Favor Of Industrial Concerns-That’s What It’s For!

The first publication of the ISO’s GUM (Guide For Expressing the Uncertainty In Measurement)in 1993 was a technical triumph that marked an important point in a long but progressive slog, dating back at least to the last days of WWII.

?

What was the context that lead to the creation of the GUM and also explains its inherent bias so easily?

?

At the end of this second, even more terrible world war, societies had to choose between continuing their wartime focus with its exciting and heightened singleness of purpose or retreating as far as they could from any reminders of the recent destruction. Of course, every society chose different proportions of each.

?

As bad as WWII was, we saw benefits from it. One benefit was that, as peace returned, a relatively large number of people awoke to the fact that some of the same tools, attitudes, and techniques that helped when we needed help so desperately in the War, would also work very well in a surprising variety of post-war applications.

?

How did this sea change reflect itself in Measurement practices that have come down to us?

?

In fact, a quite sizable group of engineers and technicians came home from their war and saw that the engineering problem of delivering on new consumer production demands required better measurements, in exactly the same way that wartime production challenges had. Some within this larger group went even further, arriving at a new understanding that the “better measurements”? requirement also?might?sometimes have to include a much more comprehensive characterization of the uncertainties that accompanied each of those critical production measurements. (I plan to return to that key word “might”)

?

At the same time, postwar Measurement Risks increased because much larger capital investments were now at stake! In August 1945, these people recognized that a Measurement Uncertainty piece was not even remotely ready to plug into the task of managing production processes, ?were a manager to decide that that the addition of that capability was critical to consumer acceptance. Engineers recognized and resolved the presence of Measurement Uncertainty when they had no other choice, but their production solutions were “ad hoc”; they took only their particular challenge into account because there was no ISO GUM. That document was almost 40 years in the future.

?

Uncertainty about Uncertainty

?

Postwar confusion was everywhere. Within Measurement practice, sometimes integrating Uncertainties into control of these industrial processes was critical for commercial success and sometimes it wasn’t. Who was to make that decision? Consumers continued to make this decision but only ever indirectly through their buying choices. The postwar transformation of the western industrial landscape was a “root cause” leading directly to the ISO GUM and revealed its industrial bias, even as todays measurement and uncertainty environment continues evolving.

?

How did a Guidance for Estimating Measurement Uncertainty end up also containing a bias?

?

It’s easier to illustrate why we had to build this bias right into the GUM if we picture the situation faced by two different process managers. One is a mid-level production manager in any industry making consumer goods. The goods could be pencils, freezers, or tractors. Let’s call this manager Jack. The other manager is responsible for managing the diagnostic reliability and usefulness of the output from an analytical lab serving a hospital and its doctors. We will call her Zoe.

?

Suppose that Jack works for the Ford Motor Company. I say that consumers indirectly drive the question of when and when not to employ Measurement Uncertainty analysis. Way back in 1920, if the public had been willing to accept Ford automobiles that failed 65% of the time in their first 15 miles, Ford production practices, including Ford measurement practices, would have reflected the tolerances necessary to deliver this level of performance and no better. Fords competitors would have gladly helped enforce this quality level either if Ford had hesitated to meet it or if Ford had pushed too hard to innovate in areas that the public didn’t actually care about.

?

What we saw, instead, was Henry Ford’s acquisition, over a century ago, of an entire Swedish firm producing some of the most advanced physical measurement tools available anywhere on the planet. I am referring to the “Jo Blocks” that Ford machinists used as their standards for measuring critical physical dimensions, so that Ford cars fit together and worked like the public wanted.?

?

In specific, Fords machinists knew that the Jo Blocks were so temperature sensitive that they could not even touch these devices bare-handed. They also knew that Ford needed to stabilize Jo Blocks in temperature controlled spaces. Back then there was no “ISO” to force Ford mechanics to learn this. They learned because they were smart and curious, and also because Ford and the whole car market gave them no choice.? 25 years later, WWII ensured that War material manufacture managers solved thousands of similar critical dependencies on the fly,?or else.

?

As a modern manager, Jack swims in a sea of data. Some of his tolerances are critical enough that for him to successfully manage them and see that success reflected in car sales, he must sometimes consider the Measurement Uncertainty within his tolerances.? Other pieces and parts are not so sensitive, and Jack definitely wants to avoid over-analysis because it’s time consuming, expensive, and might easily constrain his production output.

?

Next, let’s switch over to Zoe’s situation.

?

Her healthcare sector also changed tremendously during and after WWII. For example, the War prompted the world-wide “Baby Boom”. At every age increment since 1945, this population bubble has placed higher and higher demands on their healthcare systems. New drugs and technologies (not to mention the discovery of DNA) have revolutionized parts of Zoe's business, but one big area has lagged significantly, and that area is data collection and analysis. Why?

?

Zoe manages a healthcare analytical laboratory. This lab processes samples from various parts of the bodies of human beings who, in this setting, we are optimistic enough to call “patients”. A doctor ordered the capture of these samples before they begin heading toward Zoe. Every step in sample handling and measurement is very expensive, but one in which we invest because we hope that it supports the diagnostic decisions made by doctors about those sampled patients, in order to restore their health.

?

In our short stroll between these two roles, we encounter a huge divergence.

?

We started by visiting a production process manager, Jack. The concern and pressure that Jack constantly feels also provided enough horsepower to fund the entire huge decades-long effort that finally produced the GUM. Whenever Jack or one of his production engineering staff decides that they cannot avoid factoring Measurement Uncertainty into the way that they manage and monitor production, the GUM is there for him.

?

Yet when we glance over at Zoe’s world, even though we see doctors and their analytical lab partners, mobs of patients, and tons of capital investment, we still see nothing remotely equivalent to The GUM anywhere. Remarkably, a single person could easily be the ultimate customer of both of these complex systems in a single day!

?

What is at the heart of this divergence? What is its fundamental characteristic? To me, the key is in the “data”. The data environments that Zoe and Jack experience could hardly be further apart, but they both still produce a single thing that we call “data”. Yet the ISO GUM speaks almost exclusively to the production and control of the data and its accompanying Uncertainty found in Jack’s world.

?

Let’s dig further.

?

Over in the hospital, we can look at the nuts and bolts attached to the patients data, the analysis of which everyone hopes will help support a correct diagnostic decision. One critical feature both Zoe and the patients doctor face is that they almost always inhabit a statistical world of single samples, where “n=1”. There is no Law Of Large numbers anywhere in the vicinity to which we can appeal for assistance unless we start studying epidemics across relatively large populations.

?

Aside from its tiny size, there is a totally different but also very troubling fundamental aspect of this single patient sample. When Zoe’s lab first processes these samples, she must accept them from a person who is by definition, not “normal” even by their own unique and individual standards. This means that all the data that Zoe ever gathers in this lab are from samples that may be “statistical outliers” and have been ever since someone ordered them! Zoe may find multiple test results from a sick patient to be within normal bounds, but somewhere there is at least one test result that isn’t normal, or the person wouldn’t have turned into a “patient” in the first place. Unfortunately, it is Zoe’s function to distinguish between normal and abnormal but based on only one shot. These sometimes be life or death decisions. This is exactly the environment that prompted Nick Barrowman to write his wonderful and short six page article, “Why Data Is Never Raw”.[i]?These two fundamental data factors create a tough start for Zoe, but from there, things don’t get much better for her.

?

More nuts and bolts…

?

Beside that “n=1” challenge, and the lack of any central tendency, Zoe’s analytical lab also has other designed-in vulnerabilities. For example, the supply of powdered and liquid “analytes” that Zoe’s lab workers apply to the sample to create lab results are also variable despite their manufacturers best efforts. Successive batches are not identical! How do we task Zoe with managing quality assurance for even a single test protocol over a month when the test analyte batches ran out seven times in that period? How do we knit those broken but serial data lines together into a single string from which a doctor can draw an inference, or allow a lab manager to compare the performance of two labs? Zoe manages it, but its reliability is shakier than what Jack experiences as he manages his production line!

?

More worries for Zoe

?

If data is too ephemeral, let’s look at some labware. Some of the physical laboratory equipment on Zoe’s turf is steady and robust, and some, not nearly so much. I speak here from my experience calibrating scales, balances and pipettes in laboratories. I would tend to trust a scale. In fact, Jack and Zoe might conceivably even be familiar with the exact same mass measurement instrument! I wouldn’t trust a result from a pipette as far as I could throw it because they are so frequently abused or misapplied.? Labs use pipettes at many stages in the process of delivering their lab results. A well calibrated, maintained and appropriately selected pipette (and tip!) is a thing of beauty, trust, and reliability to a lab managers eyes. (This is just as true for the manager of an upstream drug development analytical laboratory, using hundreds of pipettes, but I would be getting way ahead of myself if I were to expand over into that countryside.) All of these vital pipette quality attributes are, however, completely invisible to any casual lab visitor or auditor, much less Zoe herself. These qualities may or may not actually be present in any particular lab pipette on the day we visit, nor at any point in a past extending backwards for a distance that we have no practical way to determine.

?

Meanwhile, back at Jack

?

In contrast to Zoe, Jack stands astride a data flow that can quite easily reach epic proportions. Let’s peek a little more deeply. Yes, this data torrent flows between banks that are progressively more perilous the closer Jack’s process approaches them, but the behavior of the process in the vicinity of any tolerance or limit is comparatively quite well known in contrast to the mysteries of human health.

?

We only begin to see some dim outlines of human health when we look in the aggregate and not case by case. But patients don’t happen to care at all about our aggregates. They show up with symptoms, not tolerances. Regardless, to the patient in that waiting room, or sitting on that gurney, they are The Case.

?

Even considering the increasing risks Jack sees when his production process approaches a limit or tolerance, we can still describe a theory of optimizing his industrial process and often we can deliver on it, too. But in contrast, the “optimal patient” is invisible because they haven’t yet defined themselves yet by arriving for their appointment. Furthermore “optimal patient” is a contradiction in terms.

?

Even this very simple exploration of the differing conditions we find in these two differing sets of job responsibilities explains why our understanding of the uses and limitations of Measurement Uncertainty has advanced so much more quickly for Jack in the industrial factory than for Zoe in the hospital. We have heard this theme of uneven development repeated many times in the History of the Measurement of Uncertainty[ii].

?

I find it hard to support a guess that the causes for the differences between Jack and Zoe’s struggle is only about money. Funding for healthcare continues to grow in a robust fashion to say the least! Can it be remotely possible that consumers have higher expectations for their front-loading washers than they do for their heart procedures?

?

Compare customer input in industrial process management with that in healthcare process management. There are huge differences! If we could select a health diagnosis like we select a freezer, there would be no human mortality at all!

?

I do know that healthcare policy is liable to have quite? bit more ambiguity than consumer expectations for an appliance. For example, no one lobbies for poorer freezer performance. My favorite counterexample here concerns tobacco. Decades ago, we settled the fact that smoking tobacco kills people despite the efforts of Sir Ronald Fisher and the industry executives who testified under oath before our Congress. Why is its sale legal still in the same country where drug manufacturers are supposed to tremble at the prospect of an FDA audit? Why do we even see new tobacco products rolling out to our kids?

?

So far, I have courageously resisted dragging the question of pharmaceutical manufacturing over into Zoe’s neighborhood, but I can resist no longer. That is because I never tire of quoting a former FDA inspector, Greg Bobrowicz, ?as he spoke to a bunch of us ASQ members from the Pharma sector: “If Toyota made cars like your industry makes drugs, you could go to their new car showroom and select from cars with somewhere between three and five wheels!” (audience gasps follow…)

?

The ISO GUM reflects our practical strategy to avoid trying to swallow an impossible analytical meal and instead, break it into smaller pieces that are just barely digestible (to lean on a health analogy). We are lucky to have it, biases and all! When we decide that we must avoid writing a truly universal attack on Measurement Uncertainty, we accept bias.

One of the topics that the GUM still avoids is one that I touched on briefly earlier. The GUM tells us “how?” but not “when?”. When should we decide to factor in the Measurement Uncertainty that we know is always there anyway, and add it to an assessment we might make about our production measurement system? We also know that Measurement Risk rises as a process approaches a production tolerance. Obviously, Measurement Uncertainty is a risk factor which contributes non-linearly to Measurement Risk, but the GUM remains mute on this linkage. Measurements are FOR something.

?

The GUM perspective has always existed in a vacuum as far as these questions are concerned even though the true customers of the GUM as a tool are surely interested in answers to questions just like these. It’s safe to assume that few people tackle the GUM as a source for light reading!

?

Thanks for sticking with me once again!

_____________________________________________________________________

[i]?https://www.thenewatlantis.com/publications/why-data-is-never-raw?HIGHLY recommended; short (6 pages) and sweet, and what’s more, it comes from the pen of a hospital statistician who can write like a dream! Nick is so clear that it should be easy for a reader to see if they agree with him (I do).

?

[ii] ?The History of Statistics: The Measurement of Uncertainty Before 1900, by Stephen M. Stigler, Cambridge, Mass. Harvard University Press, 1986. I have gone back to this excellent book many times for grounding.

?