"Numbers Never Lie", Except When They Do

We've been inundated with some variation of this phrase. We reflexively, with an almost religious fervor, sing the praises of "big data". (Note: I'm not above this.) Without critical analysis, our answer to ostensibly every issue in business is to "listen to the numbers".

Such thinking comes from a good place and has, on balance, had a positive effect. The demand for quantification is in large parts a demand for stronger arguments; assertions must be proved out and constantly tested. In some ways, thus, it's an infusion of the scientific method into the business world. No longer are truisms accepted as fact. These are all good things and have almost certainly been a driving force of the higher living standard we enjoy today.

Yet as any statistician worth their salt will tell you, it's critical to know the limits of your data and to keep your conclusions appropriately modest. Furthermore, there are many situations where, no matter how brilliant the statistical model, the truth can be complex and perhaps indecipherable. Economist F.A. Hayek made this point brilliantly in his 1974 Nobel Prize acceptance speech:

...I confess that I prefer true but imperfect knowledge, even if it leaves much indetermined and unpredictable, to a pretence of exact knowledge that is likely to be false.

Hayek was a master mathematician. But he feared that numbers were often being used to support expansive government policies based on a level of certainty about affairs that simply couldn't exist.

This insight is applicable to the business world. We often treat data as the Holy Grail without understanding limitations and context. Just because there's a number available doesn't mean that number provides any more clarity than a qualitative statement (or no statement at all). Often it obscures things by providing a false sense of mastery.

Such undue faith in numbers for the sake of themselves abounds in other fields as well. In public policy and journalism, pundits cite quick-hitting stats to purportedly "explain" insanely complicated issues like gun violence. (Two or three numbers can't do this, sorry.) In sports, a Cleveland Browns beat writer made a graph showing the probability of Johnny Manziel starting. (This was just a way to make his "finger in the wind" guesses look scientific.)

When we examine data, we shouldn't automatically accept what we see, particularly if it flies in the face of common sense. Statistical conclusions at odds with theory are often based on faulty inputs or uncontrolled-for variables.

I certainly don't advocate for the dismissal of data-driven analysis; far from it. In both my day job and my following of political issues I use and form opinions from data. Rather, I simply suggest we take a more tempered view while crunching the numbers. The best academics pepper their conclusions with conditionals and skepticism. Similarly, in business, it's not the end of the world to say "I don't know", or to replace a bogus chart with a more truthful but less precise statement.

Numbers never lie (because they're not sentient). But they can make people draw some silly conclusions.