WAGs, SWAGs and Statistical “Crazy-Making”…courtesy of the Normal Distribution

Is it too much to ask people making six-, seven-, or eight-figure salaries to gain a BASIC understanding of variation?

They would be amazed at the "Joy in work!" this could create.

I had been mentoring a very good data analyst at a large medical organization for three years.?Despite the enthusiastic support of his medical director, it had been pretty much an all-out war with the C-suite executives to implement "data sanity" – resistance, to put it mildly, had been fierce from the start.?

I received a note from him:

"I'm sorry to report that it appears control charts [of key indicators] are nearly dead... As of last week, they have been pulled off all but one report...

"In other news [the organization] has moved towards Lean Six Sigma.?The first Black Belt 'course' is being offered right now – and I have been 'drafted' to teach the statistics portion... I have been working on my slides over the past couple of weeks – and I must say that I don't understand where any of this is going to come in handy for Quality Directors...I spent half of an hour trying to find information on calculating the confidence interval for the correlation coefficient by hand.?It involves the inverse hyperbolic tangent function... I'm sure everyone will get that one, right?!?

"It all seems a little ridiculous to me."

It's also ridiculous how the resulting “certified” people then subsequently perform inappropriate analyses that inevitably lead to serious consequences on hard-working people – when wild-ass guesses (WAGs) and/or their conversion into statistical wild-ass guesses (SWAGs) are used to take important actions.

I Cant Make this Stuff Up

Published rankings with feedback are very often used as a cost-cutting measure to identify and motivate (perhaps fire?) "those bad workers." Some are even derived, er... uh... statistically?

In an effort to reduce unnecessary expensive prescriptions, a pharmacy administrator developed a proposal to monitor and compare individual physicians' tendencies to prescribe the most expensive drug within a class. Data were obtained for each of a peer group of 51 physicians – the total number of prescriptions written and, of that number, how many were for the targeted drug.

Someone was kind enough to send me this proposal – while begging me not to be identified as the source.?I quote it verbatim (my emphases).

Given the 51 physician results:

“1. "Data will be tested for the normal distribution,

“2. If distribution is normal – physicians whose prescribing deviates greater than one or two standard deviations from the mean are identified as outliers,"?[DB:?SWAG]

“3. If distribution is not normal – examine distribution of data and establish an arbitrary cutoff point above which physicians should receive feedback (this cutoff point is subjective and variable based on the distribution of ratio data).”?[DB:?WAG]

For my own amusement, I tested the data for normality and it "passed" (p-value of 0.277, which is > 0.05).?Yes, I said "for my own amusement" because this test is moot and inappropriate for percentage data like this (the number of prescriptions in the denominators ranged from 30 to 217)...but the computer will do anything you want.

The scary issue here is the proposed ensuing analysis that will result from whether the data are allegedly normally distributed or not.?If data are normally distributed, doesn't that mean there are no outliers? But suppose outliers are present – doesn't this mean they're atypical? In fact, wouldn't their presence tend to inflate the traditional calculation of standard deviation? But wait, the data passed the normality test... it's all so confusing!

Yet that doesn't seem to stop our quality police from lowering the "Gotcha!" threshold to two or even one standard deviation to find outliers (in my experience, a far more common practice than you might think).

Returning to the protocol, even scarier is what's proposed if the distribution is not normal: Establish either an arbitrary cutoff point, usually a WAG for either (1) what the administrator feels performance "should" be or (2) the point that will catch a pre-determined arbitrary percentage (ending in “0” or “5,” of course) of “bad” performers and/or reward a similar arbitrary percentage of alleged “good” performers.

I'll play his game.?Because the data pass the normality test, the graph below shows the suggested analysis with one, two and three standard deviation lines drawn in around the mean – green, yellow, red interpretation of course!.

The standard deviation of the 51 numbers was 10.7.

Ouija board, anyone?

And the Consequences?

Depending on the analyst's mood and the standard deviation criterion subjectively selected, he could claim to statistically find one – or 10 – upper outliers. Just curious: how does he intend to deal with the performances below the one standard deviation limit of 5.15 percent…and the three zeroes?.?

Even worse, he could have just as easily used the WAG approach and decided that 15 percent was what the standard "should" be – resulting in feedback to the 27 physicians above 15 percent.?

Or maybe he could even set a "tougher" standard of 10 percent, in which case thirty-five physicians would receive feedback. This feedback consisted of a wealth of educational material on “gold standard” practice.?When I present this example to a roomful of doctors, the room erupts in laughter. Then when I ask what they do with such "helpful" feedback I see a beautifully synchronized collective pantomime of throwing things into the garbage.?

There is also the common alternative WAG: Lets go after – oops, I mean give feedback to – the top quartile (or top 10%...or 15%...or 20%.)

What's not so funny:?similar WAGs and SWAGs are fast becoming techniques in the current pay-for-performance craze in healthcare.?Who knows??Maybe some of these schemes may even involve the inverse hyperbolic tangent function, so my mentee's training will not have gone to waste.

As he said, "It all seems a little ridiculous to me."

These and similar analyses are all-too-typical and usually result from misapplication of the "superstitious nonsense" people have been taught in statistical belt courses.??How much waste in time, money, and morale do analyses like these cost you? (A lot!)

In everyday improvement, normal distribution superstition causes far more confusion and problems than it solves.

Have I mentioned that I never teach it? (and I'm an MS degreed statistician)

The (simple) alternative? Here is a follow-up article.

====================================================

Chapter 7?of my book Data Sanity thoroughly covers the technique Analysis of Means (ANOM) needed for this example and is one of the very few?available resources for learning it. ANOM is a very powerful statistical stratification technique that has become one of my major analyses. Unfortunately, it seems to remain a well-kept secret and woefully underutilized, if at all.

Data Sanity: A Quantum Leap to Unprecedented Results is a unique synthesis of Dr. Deming's philosophy emphasizing?the sane use of data, culture change, and?leadership principles as a roadmap to create a road map for excellence.

• It is available on US Amazon
• For UK readers, it is available on Amazon UK

Please see my LinkedIn profile for more information and clarifying downloads regarding?Data Sanity – Introduction/Preface and brief chapter summaries.

?