3 Simple Ways To Reduce The Risk Of Curve-fitting in Trading Strategy Development
Curve-fitting is almost certain death for a trading or investment strategy.
So, what is curve-fitting?
Well, you know when you test a trading or investment hypothesis, fall in love with the historical results, and then the idea fails to generate similar (or even positive) returns once you decide to trade it live?
Most of the pitfalls of system trading, and trading in general, can be avoided or mitigated following these three simple techniques or rules of thumb.
- Use out of sample data! Out of sample data is simply withholding some of the data in your “test” period for further evaluation. For example, you have ten years of historical data and opt to put the last 30% in your back pocket. You develop a great trading strategy on the first seven years of the data set and then whip out your “out of sample” data (remaining 30% from your back pocket) and validate your findings. If the strategy fails to produce similar results in the out of sample data then you can be almost certain you have curve-fit to the first seven years of your data set. Below is a chart of a strategy built using Build Alpha that highlights the out of sample period. You like to see similar growth (and characteristics) in both the in-sample and out-of-sample data.
2. Make sure your strategy has enough occurrences or trades. This can be simply explained with a coin flip example. If you flip a coin 10 times and it lands on heads 7 times you cannot be certain you do or do not have a rigged coin. However, if you flip a coin 10,000 times and it lands on heads 7,000 times you can have very high confidence it is indeed a rigged coin. In trading, if a strategy has 30 trades then it is unlikely you would have high confidence that what you have found is legitimate. However, if a strategy holds up over 1,000 or 3,000 plus trades then you can have higher confidence you’ve discovered true edge.
Below is a photo of only 30 coin flips and below that is a photo of six different trials of 100,000 coin flips. You can see after a large number of occurrences things tend to converge toward the true expectation (also known as the Law of Large Numbers).