On The Notion Of Over-Fitting
Recently, on a Quantopian thread, the debate touched over-fitting stock trading strategies. I tend to have a different point of view on this subject than most since I find the very definition of strategy over-fitting as something that describes very little of what is. It resulted in the following post which is replicated here.
The notion of over-fitting a trading strategy might technically turn out to be quite a misnomer. It might require to do so many compromises that its very notion becomes almost irrelevant.
I see the problem of strategy over-fitting as two separate, sequenced and interacting problems: first, a stock selection process, and then a trading process.
The selection process is like a stock screener of some kind. A way of selecting for some reason or other some stocks to trade on. In my post on gaming multiple strategies, it was said that selecting 500 stocks out of some 2,000, like in the QTU stock universe, had 5.6^486 possible combinations. Every day, every week, or every month you would have such a selection to make from this huge stock universe. This number is so large that even a slight shift in criteria could yield quite a different list of stocks, each with their own idiosyncrasies. You could schedule your trading procedure 30 minutes later and get different answers while during that time global activity on the planet would not have changed that much.
On the other hand, the specifications could be so specific that a single list might be the only thing that comes out.
The Selection Process
The outcome of the selection process is what will be fed to your dynamic trading process. It is therefore critical to understand which stocks are being picked and for what reasons.
Is it over-fitting if you change the selection process resulting in a new selection that might not be comparable to the previous one? We could all argue that that would be sufficient to get different answers even if none of the trading procedures or program logic were changed an iota.
Over-fitting implies that you have something “fitted” to compare it to (from where you derived the expression over-fitted) or that you have taken enough samples from the available selectable universe to reach the notion of averages. How many samples would that be? One might not be enough!
The over-fit notion might possibly apply if we had averages where we could say: above or below those averages. But, there, the problem is that we have no means to locate or identify what the averages might be. The selectable universe is simply too large. You take a million samples (which requires running a million simulations, tally all the results) and all you have is 10^6 out of 5.6^486, leaving 5.6^480 untested scenarios.
Changing the portfolio selection process does not make it over-fit, just different.
It is one combination out of 5.6^486. And, a few days later, the whole selection process will need to be done again, and again be a 1 in 5.6^486. It would be sufficient to change a single factor to get a different selection.
To grasp the magnitude of 5.6^486 take 10^10 supercomputers (that's 10 billion machines) times 10^16 operations per second times 10^14 seconds over some 10 million years of trading. This adds up to 10^40.
NO Exhaustive Search
So yes, nobody has yet to make an exhaustive search, Monte Carlo style, or otherwise. Not even close. One million more of those supercomputers raises the total to: 10^46. Still, an infinitesimal fraction of the total (it is: 0.0 followed by 439 zeros, and then a one). I think you see the point. The selections we make could be pretty unique and come with no information of the world they come from (mean, standard deviation, and what have you).
How could we declare something over-fit if there is nothing that we can compare it to or somehow declare as a representative average of this huge set of possible scenarios? Even the indices we use are just 1 in 5.6^486 other possible choices. So, we make compromises and simply declare, this whatever something, as an average of what is out there.
Whatever you select as your trading universe will most probably be different from everyone else unless they use the exact same trading program you do on the same data from the same data provider and did those simulations under the same conditions using the same trade times over the same time interval.
Survivorship Bias
The best you could do is bias your selection toward survivors. The financial literature recommends: avoid survivorship biases as much as you can. But, that is exactly what you should do: seek out survivors by all means. You want to trade the best that is out there, not just anything that comes out of this huge universe of possibilities.
Your selectable and tradable stocks usually should not be penny stocks or those on the verge of bankruptcy, even though you could specialize in those. Going forward, you will want to play only survivors anyway. Those that have positive future prospects of doing more business. None of the nearly bankrupt candidates should ever be able to stay on your tradable stock list. If it is a bias to want to play the best stock candidates you can find, then so be it, do it. It is just plain common sense. And it is not because you are programming something that you should lose it.
However, the caveat is: everybody else is trying to do the same thing (the millions of them). And guess what, a lot of them navigate toward the same solution. For instance, all the traders that sort their tradable universe by highest market cap should all get the same answer, all the time.
There was only one possible past universe to select from and it is of historical record.
Therefore, whatever their respective trading strategies, they will all be a variation on the same theme which is: how can you take advantage of large caps sorted by size?
Just by putting this observation on the table and the problem changed.
The Problem Changed
Now we have traders (he/she) playing a variation on a theme. Whatever trading strategy they design, it will deal with an ordered list of the highest caps, the exact same selection set as the other guy. For whatever, when you analyze a lot of strategies, it is what you find. And this reduces the problem to a single set in this huge stock universe... which furthermore becomes the same for everyone on this theme. Is this an over-fit? Again, I don't think so. It is just a selection process.
I presume, except for published strategies, that everyone has a different selection process making their tradable stock universe unique. And from day to day, that universe could change. It is not from the stock selection process that we could declare a trading strategy in some over-fitted territory. All you have is a selection, whatever it is. And whatever criteria served in that selection process can also be applied going forward.
The problem is different going forward since you will not know what your selection process will choose. It is not a question of over-fitting, nor is it a notion of random-like selection. Going forward, your strategy will select its stocks, but you will not be able to know which factor dominated, or why next year at this date a particular stock made the list. You cannot even know if it will be there or not. However, some do have much higher probabilities of being there than others. Does your trading strategy suddenly become over-fitted because your stock was there or not there? Again, it is not from the selection process itself that we could speak of over-fitting. For the trader, that stuff did not make any money. Or did it?
A long-term investor would argue that it did and that the selection process was the most important of all. It should be observed that you do not see traders in the Fortune 500, only bag-holders that have held on their shares for years and years, which in a way defends their argumentation.
As for the traders, their respective trading strategies will differentiate them.
Independent Computer Software Professional
6 年The problem shrinks so much that now you have not only a finite and relatively small universe (a few thousand stocks), but also a unique outcome. And to top it off, this universe of sorted market caps will be the same for everyone using that selection criterion. On this selection, or anything comparable, meaning that you do a selection under any criteria, we could start to view the notion of over-fitting among a specific group having used that specific selection criterion, average out their results, and determine which are performing better or worse than the average for the group. However, what will differentiate players in that group will be other strategy criteria than the selection process itself since all had the same. It will be their respective trading methods that will differentiate them. And to add to the uncertainty, we know nothing about all those groups (portfolios) or their respective trading methods without monitoring them all, which again is beyond our technical abilities.
Independent Computer Software Professional
6 年The problem shrinks so much that now you have not only a finite and relatively small universe (a few thousand stocks), but also a unique outcome. And to top it off, this universe of sorted market caps will be the same for everyone using that selection criterion. On this selection, or anything comparable, meaning that you do a selection under any criteria, we could start to view the notion of over-fitting among a specific group having used that specific selection criterion, average out their results, and determine which are performing better or worse than the average for the group. However, what will differentiate players in that group will be other strategy criteria than the selection process itself since all had the same. It will be their respective trading methods that will differentiate them. And to add to the uncertainty, we know nothing about all those groups (portfolios) or their respective trading methods without monitoring them all, which again is beyond our technical abilities.
Independent Computer Software Professional
6 年Alexander, I like your argumentation. However, whatever cross-sectional selection process we might want to use, it will be one combination taken from the whole universe of selectable stocks and selection methods. We would be hard-pressed to say that a particular selection is representative of the whole not having scanned the whole selectable universe of choices since technically it is not even achievable in a billion lifetimes. It is why we make compromises and pick a method, some method, and then deal with it. That we make a specific selection based on whatever criteria we want, that selection might be so unique that no one else on the planet might have made the same choice but it would still be one combination out of this humongous universe of possibilities. On the other hand, your selection criterion could be unique. For example, order all stocks by market capitalization.
Quantitative Developer
6 年I don't think combinations are the right way to view the universe selection process. Instead consider cross sectional notional allocated (vs capitalization, for example, or perhaps other dimensions you find useful). This analysis should be done separate from what you traded (tradable vs traded). In this way it's possible to find methodologies with consistent tradability criteria that eliminate survivorship biases on some reasonable time horizon. Admittedly this does kind of push the problem off on to the "traded" part of the process, but that makes sense as you should only really be allocating capital in places you actually feel you have an edge/prediction (alpha), or have some other purpose (risk/hedging). I guess I might reframe the interesting part of the problem as: "does alpha exist consistently across our universe": we all know the answer is no. On the use of the term over fitting. It's not insane to use the term if you think in the classical statistical definition of a loss function vs a free variable and the inevitable data-mining in the industry. I mean, no one thinks they have representative data of the entire support when doing quant work. Is it overused? Sure, but it's a recognition that we're in sample.