Beware the Tides of Bias

I want to find the average height of the American male. I’ll travel the country, tape measure in hand, to seek out every man in America. Yeah…no. I don’t have the time or resources to do that. I know what I’ll do. I’ll take a sample! I’ll choose five men and measure how tall they are.

The way I really should take that sample is to gather all the men in America, dump them into a big barrel—like they do with raffle tickets, roll that barrel several times to mix them up good, and pluck five out at random. Dang, I don’t have a big enough barrel. But I just happen to be going to a basketball game tonight, so I’ll traipse into the locker room and measure the five starting players of the Detroit Pistons.

Do you see what I did there? Whether intentional or not, I just biased my results. A little casual observation will quickly confirm the average height of the five starters for the Pistons—six-foot seven-inches (that’s approximately 2 meters for all you readers living outside the American borders)—is about a head above most other men.

That kind of bias is pretty evident, but what about bias that’s not so obvious? Things like a mixed batch of material where half happens to be at the high end of a specification and the other half is at the low end. Or an unanticipated factor is present in an experiment and it affects how other factors perform. Or a gauge shifts out of calibration.

Bias is simply an effect which changes the measured result from its true result. If it’s obvious, then prior to gathering data it can usually be anticipated and corrected. If it’s not obvious, then it creeps in undetected and alters your data. It’s insidious because you have no way of knowing your data was affected. For example, take that raffle ticket barrel mentioned above. It’s being delivered to the hall where the raffle will take place. Along the way it rains and some water seeps into the barrel. You dump tickets in there, unaware a small puddle has pooled in the barrel. The first tickets dropped in get soaked and cling to the side of the barrel. Other tickets landing on the wet ones absorb diminishing amounts of moisture until there are dry tickets resting on the others. When the barrel is turned, the wet ones stick to the side and only dry tickets mix in a random fashion. When the winner is drawn, the results are biased because the tickets stuck to the side never even had a chance to be picked.

Bias is like waves on an ocean. When the water is calm, you can easily navigate a small dinghy with no issues. When the waves are mild, or even moderate, a sturdy ship may rock a bit, but still navigate fine. However, if the waves reach the height of the boat, or worse yet above, then there is grave danger the boat may capsize. And, if a tsunami rears its ugly head, then the ship is lost.

Like a sturdy ship sailing in mild or moderate sized waves, statistical methods are generally robust enough to counter the effects of a little bias—after all, the methods do involve incorporating process variance into the calculations. At its worst, a tsunami of bias can demolish the integrity of your data and produce false results, leading you to erroneous conclusions based on interpreting what you believe is true. Your ship sinks.

What can you do? Understand bias happens. Ignorance keeps you unaware. Understanding allows you to start developing strategies. Realize study designs are a team effort. Invite other process Subject Matter Experts (i.e., SMEs) to the party. Sharing tribal lore is one of the best ways to gain awareness of potential sources of bias. If your study involves a multi-step process, then make sure to include SMEs both upstream and downstream of your little oasis. SMEs upstream may help point out unanticipated interactions with what they’ve already done. SMEs downstream not only need to be aware of what your doing, but they may also help uncover potential sources of bias because they have experience dealing with what comes to them on a regular basis. In addition, don’t forget the statistics SME. Statisticians are trained to recognize everything has variation which could contribute to bias. So be patient as they work through the gory litany of sources of variation. It may be your process knowledge gives you high assurance some of those sources are negligible. Still, it’s good to consider all sources in case you’ve simply overlooked one that may not be negligible.

I recall working with an engineer on an experiment to test a current system output against a proposed change to improve a characteristic. The change required half a day to set up, so to save time the engineer suggested making the change on one piece of equipment and leaving the current system intact on another piece of equipment. It was a reasonable suggestion. I asked one question, “Do you see any difference in measurements on that characteristic between the two machines?”

The engineer said there was a known difference between the two, paused in reflection, then wide-eyed said, “Oh, I see what you mean. If they measure different now, then I won’t know if it was the change that affected the process or just the usual difference we see.”

It was simple and obvious after I pointed it out. This is not a slam against the engineer, who was brilliant, but it demonstrates the tunnel-vision that can occur when designing a study only in the vacuum of your own mind. Study designs are a team effort.

The engineer ran half the experiment as originally planned, then took a half-day to reverse the setup and finish. Our collaboration successfully eliminated one source of bias from ruining the data. We could then assess the effect of the proposed change on both systems.

The process of elimination is an amazing tool to use in any problem-solving situation. Make it your friend. Generally, if there is an expected result in a process, then confirm with objective evidence to eliminate it as a source of extraneous variation. Otherwise you have a prime candidate for bias. It doesn’t have to be a full-blown formal study. Maybe more of a “sanity check.” Confirming with objective evidence includes any process where two or more “things” should behave the same (e.g., different assembly lines or machines producing the same product, multiple gauges measuring the same characteristic on items produced on those different assembly lines or machines, product received from several suppliers). For manufacturing processes, a program of Statistical Process Control (SPC) with Control Charts can provide that evidence.

Okay, so you gathered your SMEs, brainstormed until all your brains were stormed out, confirmed with objective evidence all the items that piddled out of your brainstorming sessions, and worked through every meticulous detail. Now what? Well, if you’re a praying person, you pray. If not, then hope. All this rigor and we’re still back to hope. Despite all your efforts and careful planning, bias can still creep in. So, if the results are different from what you expected, especially if there is ample engineering understanding and laws of physics to back up the expected results, then do a post-mortem. Try to figure out what unanticipated source of variation biased your study. It could turn up the elusive missing link and increase your fundamental understanding. You’ll have to redo the study, but that may be a fair price to pay in order to reduce the effects of a previously unknown source of bias. Hence, you’ll get a cleaner signal of how your process behaves.

You were warned by the Bard to “beware the Ides of March.” Consider yourself warned to beware the tides of bias.