How Leaders Improve Their Decision-Making with the Brier Score

Verification of weather forecasts has been a controversial subject for more than

half a century…. One of the most important difficulties seems to be reaching an

agreement on the specification of a scale of goodness for weather forecasts.

Glenn Brier

Monthly Weather Review

February 10, 1950

US Weather Bureau

Australian meteorologist W. Ernest Cooke was the first weatherman to assign

probabilities to forecasts in 1906.

It didn’t catch on.

Another 35 years passed before leaders demanded meteorologists assign

probabilities to forecasts. They were demanded because the stakes were high.

Americans and the Allied Forces were at war and needed to bomb places more

accurately. Unpredictable weather between Britain and Germany required new

thinking to drive better results. As generals strategized about how and when to

bomb targets, weather and visibility was a top concern.

“Partly cloudy and a chance of rain, sir!” was as unhelpful to military leadership

then as it is to you today.

For better decision-making, leaders needed forecasts.

Leaders also needed probabilities—statements of confidence.

While World War II advanced meteorological forecasts, weathermen remained

hesitant to cite probabilities. These men of science, finally gaining recognition on

national television for their lifelong atmospheric nerdery, felt little need to risk it all

by asserting probabilities on something as unpredictable as weather forecasts.

Enter our hero, Glenn Brier, and his mission to identify a “scale of goodness for

forecasts.”

Brier Score 101

Brier, a weather statistician and academic, developed an equation to measure the

distance between what someone forecasted and what actually happened.

Brier Scores are like golf scores: Lower is better.

A Brier Score of 0.10—assuming you have collected dozens of forecasts—

indicates you’re the second coming of Nostradamus. If you maintain a Brier Score

of 0.50, it means you share the same accuracy with a dart-throwing chimp,

because it’s all a shrug-of-the-shoulders, fifty-fifty probability.

Extreme predictions (“0% chance you’ll close that deal with Lennar by December

1,” or “95% chance Deion Sanders will win 10 games this season at Colorado,”)

are rewarded—and punished—accordingly.

How to Use the Brier Score

Brier’s goal was to develop better pattern recognition and improve accuracy of

decision-making. As a leader in your business, you should share the same goal.

Here’s how you can begin to use the Brier Score.

There are four simple steps—and one tool.

The four steps are:

1. Document your predictions.

2. Assign a confidence interval.

3. Systematically review their accuracy.

4. Adjust your predictions accordingly.

The one tool is the Brier Score template.

We converted it into Excel.

Download it here for free.

(Password: hartmann)

In their book, Superforecasting, by Philip Tetlock and Dan Gardner, they write,

“Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular

ways of thinking, of gathering information, of updating beliefs. These habits of

thought can be learned and cultivated by any intelligent, thoughtful, determined

person.”

Anyone can—and effective leaders must—intentionally develop better decision-

making skills.

I’ve been using the Brier Score for the last year to improve the “scale of goodness

for forecasts” for myself as well as our executive coaching clients.

What’s been remarkable—and obvious in hindsight—is how often we make mental

calculations regarding probabilities and confidence levels, and how rarely we

document them.

To improve decision-making, leaders must document a meaningful sample size.

My clients and I generally agree to get to fifty forecasts on strategic business

efforts within the first month.

History is littered with people armed with ignorance and overconfidence. It’s a

dangerous combination.

You can reduce the risk of both by developing your own Brier Score. It will help

you make better decisions.

A world war was required to nudge leaders into demanding better forecasts. Don’t

wait for such a dramatic event to nudge you to do the same.

Thanks for reading.

Download the Brier Score template here.

(Password: hartmann)

Click the link above to listen and share this post as a podcast.

Click this video to hear about how you can use the Brier Score to improve your

decision-making.