Fooled by the System

Understanding systems also involves avoiding being fooled. I have an illustration of this I’ve presented to seminars called “Dr. Brown’s Miracle Elixir,” a magical drink that makes you feel better, no matter what kind of ailment you have. Here’s how it works. The experience of most common ailments or chronic conditions varies day to day, with the person having good days and bad days. The directions for Dr. Brown’s Miracle Elixir say that you should take it on bad days, and it will make you feel better by the next day or at the latest, within two days. In the seminar I show graphs of data of improvement, and sure enough, 76% of people taking it improve by the next day and over 90% improve by the second day. Before everyone in the seminar can click over to Amazon to purchase the stuff, however, I reveal that all of the data is randomly generated. The daily feeling level of each of the fictional subjects in the “clinical trial” is a random number between one and ten. The overall average is of course, five. So when the person feels below a five, they take the elixir.

The magic, however, is not in the elixir, it’s in a phenomenon called reversion (or regression) to the mean. Over time, the randomly generated series of numbers will tend to move toward the mean. So when a value occurs below the mean, the most likely occurrence afterward is for the numbers to trend upward; when the value is high, the most likely occurrence is for the numbers to trend downward, in both cases moving toward the overall mean. Since we carefully instruct the customers to take the elixir only when they are feeling bad, that is, below their average, it is highly likely that their feelings will trend upwards afterward. Of course the elixir is useless—how they feel is likely to improve after bad days, elixir or no. The placebo effect of taking the elixir contributes even more to the perceived effectiveness. Countless millions of dollars are made selling supplements of all kinds based on this idea. As you might notice, even a double-blind clinical trial will show these bogus results. Only the addition of a control group that takes no elixir, or better yet a placebo elixir, will show that the elixir has no effect. Systems thinkers must become familiar with the ways that systems can fool people into believing in causal effects that simply aren’t there.?