Cognitive Biases in Investing: Hindsight Bias

What is hindsight bias?

Hindsight bias is a common psychological tendency that allows people to convince themselves after an event that they accurately predicted it before it happened. This can lead people to assume that they can correctly predict other events. Hindsight bias is studied in behavioural economics because it is a common failing among investors.

Where does hindsight bias occur a) in life and b) in finance?

In life:

This phenomenon is prevalent at sporting events. After a football game, the vast majority of spectators will say they knew all along that their team was going to win / lose, despite experiencing a high degree of uncertainty during the match.

In finance:

Anyone reading the economic forecasts for 2007 today will be surprised to see how positive the outlook for 2008 to 2010 was at the time. One year later, in 2008, the financial market imploded. Asked about the causes of the financial crisis, the same experts today respond with a stringent story: expansion of the money supply under Greenspan, loose mortgage lending, lax capital adequacy regulations and so on. In retrospect, the financial crisis seems perfectly logical and consequential. And yet not a single economist - there are about a million of them worldwide - predicted its exact course. On the contrary, seldom has a group of experts been so much in the way to the backward-looking error / hindsight error.

The backward-looking error is one of the most persistent errors in thinking of all. It can be aptly described as the 'I've-always-known' phenomenon: In retrospect, everything seems to follow an obvious necessity.

Why is the look-back error so dangerous? Because it makes us believe that we are better forecasters than we actually are. This makes us arrogant and tempts us to make wrong decisions.

But one more tip, more from personal than from scientific experience: Keep a diary. Write down your predictions - on politics, career, body weight, the stock market. Compare your notes from time to time with actual developments. You will be amazed at how bad a forecaster you are. And: read history in the same way. Not the after-the-fact compact theories. But read the diaries, newspaper clippings, minutes from that time. That will give you a much better sense of the unpredictability of the world.

How to avoid hindsight bias

At VARUNA we avoid hindsight bias by using a robust framework for researching new strategies that uses sound statical principles, leaving no space for human driven biases.

Every model is always evaluated on completely new unseen data. We tend to split data into three sets, training, test, validation. The validation data is used as the ultimate voice to keep or discard an idea. Many financial studies, and many trading strategies built by several researchers, keep optimizing parameters until they obtain the perfect parameter set that leads to nice historical performance. This hindsight bias if often referred to as overfitting. With enough moving parameters one is bound to find a model that fits the data in a perfect manner. We avoid this by retaining some data as to be “fully unseen” during the research process. We also prefer models with few or no parameters at all that can be optimized. Optimization over historical data is often a source of bias.

For example, when we select the stocks to trade in a specific market, we do not just pick the ones with the highest returns across a selected historical timeframe but rather we use a universe selector. In our universe selector we split the data in evaluation, EP, test, and out-of-sample periods. We rank stocks in the evaluate period according to certain features and use the test period to verify if the ranking predicted using the features over the evaluation period allows us to shortlist N stocks that result in better statistical performance over the test period. Finally, we use the model calibrated during the fit to predict the performance over the Out Sample Period. If the top N symbols selected using the selector with parameters EP, N consistently show better performance than the average performance than the unselected universe, then we can conclude that the universe selector is indeed applicable to the strategy, which will benefit from trading the Top N symbols forecasted to exhibit higher performance.

This is just an example of how, at VARUNA, we focus on statistical evidence, leaving no space for human driven biases.