Absence of evidence and evidence of absence: The Difference between the two is growing

Taleb's latest book, "The Statistical Consequences of Fail Tailed Distributions", (I have attached the book) is more entrenched in Maths & Statistics, but sieving through it leaves us with some profound understanding of the real world we are in.

The Gaussian world of normal distribution takes about 99% of everything we try to apply concerning uncertainty, but in the real world its applicability could be as low as 1%, at least in finance, risk, or pandemic around us, that is the case.

Gauss himself as the story goes (I heard it from the tour guide in Gottingen University, where Gauss taught Mathematics), that at the age of five he calculated the addition of 1 to 99 by simply finding out how many times the 100 comes up as we add 1 with 99, 2 with 98, 3 with 97 and every time it 100 only. His mind was steeped in symmetry, no wonder he came up with the Gaussian Distribution which will live forever.

The returns on S&P 500 or the Sensex follows Power Laws, not the Gaussian distributions we grew up with, although the evidence is all around us that only a handful of securities give the returns that swing the entire field not the returns over a large number of diverse securities.

Taleb has built his empire of Extremistan (where power laws hold good) as different from the Mediocristan (where the Gaussian distributions hold good) right from his first book, "Fooled by Randomness", followed with "Black Swan", "Anti-Fragile", "Skin in the Game" and now this latest one makes the foundational principles deeply entrenched in mathematical proofs.

His examples are as striking as his final inference on most topics, but we could gain from the insights of some simple few to help us wade through the risks we try to take; the real giants of trades never take any risk, they remain the most risk averse, from Warren Buffet to the most successful trader on the floor.

His example of a normal distribution (that of heights of individuals) and asking the question what is the most likely heights of two individuals if their combined height is 4.1 meter and the answer would be 2.05 meter. Whereas if two individuals are selected from a power law distribution of wealth and the combined wealth is $36 million, the most likely wealth distribution of the two would be $35,999,000 and $ 1,000.

It is similar to asking the question that if we randomly select two securities from S&P 500 and the combined return of the two is $36 million, what is the most likely distribution of wealth, it would follow the power law and the answer would be $35,999,000 and $ 1,000.

So in effect we are saying that in a Gaussian distribution to find an outcome that is 1-sigma from the mean we would get 68% of the population data conforming to being 1-sigma away. We will see that as the distributions wander away from the Gaussian to the Power Laws, the probability of an event staying within one standard deviation of the mean rises to between 75 and 95 percent. It will be extremely rare to find a 6-sigma deviation in a Gaussian distribution, the probability of finding that outcome would be infinitesimally small, 0.00034%. These are thin tailed distribution as we call it as opposed to a Fat tailed or thick tailed (either right tailed or left tailed), where a single highly deviated outcome from the rest sways the distribution either to the right or the left. This is easy to understand with his famous Bill Gates wealth example. If you have a village of agricultural workers and their wealth is distributed and enters Bill Gates in that village, the entire wealth distribution moves to a fat tailed one as neither the mean nor standard deviation means anything as all observations other than a single one sways the result in one very different direction. Could you predict anything with such a distribution by using sampling methods?

First of all in such a distribution the law of large numbers will not hold good, nor would the central limit theorem. You would need almost hundred percent of the population data to get to the representative sample, and even that sample would make no sense of what the whole population is all about.

In a Gaussian distribution, when a sample under consideration gets large, no single observation can really modify the statistical properties. In Power Law distributions, the tails (the rare events) play a disproportionately large role in determining the properties.

Think of all the last bank runs and financial crises and you will notice that only one single event created the securities and bank assets to collapse; Goldman Sachs in its history of 136 years never had to bailed out till had to be done in 2008, on one single event of market crash from the CDS meltdown.

The entire Gaussian world of statistics built around mean and standard deviation does not apply in these distributions, neither does mean square regression rules. We would have to wait for all data to be captured to be able to say anything about the distribution, which could be difficult to interpret and take action on.

This is the fundamental problem with pandemics that from any trend it is never amply clear what the next set of data should look like; a single presence of a large deviation that has not shown in the recent data could sway the entire inference; statistical significance becomes reduced to a rhetoric.

Thus the difference between absence of evidence and evidence of absence becomes larger in such events, we should therefore be careful to make any estimates for what the future would look like.

So what does this all leave us to. Insurance of risk when there are large number of people buying insurance makes great sense as the probability of single large events remain infinitesimally small.

But we must understand where this applies and where it does not. At least in distributions with power laws we must avoid the same generalizations.