Tom Brady, Benoit Mandelbrot, and Statistical Probabilities

At some point during the LI Super Bowl somebody estimated that there was more than 90% probability of the Falcons winning. I believe the peak was at 97%. I think at this point the score was 21:3. Of course, this was based on past results. When a team was leading by that much with less than two quarters remaining, their success rate was more than 90%. And then the unthinkable happened. Tom Brady and the Patriots came back and won the game the way nobody has done it before. It was a true Black Swan event. We have a great learning moment here: statistical probabilities based on normal distributions can be very, very misleading. The reason is that many things out there in the real world don't have normal distributions. Pareto found out long time ago that income and wealth are not normally distributed. Benoit Mandelbrot ( the father of fractal geometry) studied cotton prices and found out that they didn't follow a normal distribution either. There were too many extreme price changes that completely distorted the good-looking Bell curve. Variance and standard deviations were changing all the time. From his study of cotton prices a whole new branch of mathematics was born: one that studies "roughness". It does not view irregularities and extreme events as imperfections that need to be excluded to see the "true" patterns. It accepts them and finds a way to describe them.

In statistical analysis very often the extreme events are "smoothed" or considered outliers and get excluded from analysis. This allows the analyst to use standard methods for analysis that assume normal distributions and constant variances. But this is not how the real world works. In the real world there is Tom Brady. In the real world there could be a lot of small earthquakes that more or less can be described by a normal distribution and then there are a few huge ones that are "off the charts", literally. Seismologists know this. They have a mathematical formula that shows the number of earthquakes varying by a power law: the small ones are common while the big ones are rare. Electoral forecasters don't know this. They forecast 70%+ percent probability of Clinton winning. And then Trump wins. I kind of hoped that the failed electoral forecasts will give people a pause and they will reconsider how forecasts are done. Didn't happen. We had basically the same people using the same flawed forecasting techniques making forecasts that failed even more spectacularly. Can anybody remind me Einstein's definition of insanity?