In and out of sample testing, what is it and why do it?

What is in and out of sample testing?

In-sample testing and out-of-sample testing are two methods used to evaluate the performance of a trading strategy.

In-sample testing

In-sample testing involves testing a strategy on a set of data that was used to develop and optimise the strategy.

In-sample testing is used to evaluate the performance of a strategy on a set of historical data that was used to develop and optimise the strategy. It helps to identify any flaws or weaknesses in the strategy and can be used to optimise the strategy's entry and exit parameters.

For instance, if the data set you are testing on covers 40 years, you would complete your optimisation on the first 30 years. The last 10 years would be used for out-of-sample testing.

When running an optimisation on in-sample data, the results can be overly optimistic, as the strategy has been optimised to perform well on the specific data that was used for the testing.

Out-of-sample testing

Out-of-sample testing is used to evaluate the performance of a strategy on a separate set of data that was not used during the development and optimisation process. This helps to determine whether the strategy would be able to perform well on new, unseen data (the 10 years of historical data mentioned above).

The results of out-of-sample testing are typically considered to be more realistic, as the strategy has not been optimised to perform well on this specific data set.

In theory, it is essentially operating the strategy on a set of data that it has never seen before.

Why do you use in and out of sample testing?

Ultimately, we utilise in and out of sample testing to generate a more robust trading strategy. By using in-sample testing, you can optimise the strategy and identify any flaws or weaknesses. By using out-of-sample testing, you can evaluate the performance of the strategy on new, unseen data and determine whether the strategy is likely to perform well in the future.

By combining the results of both in-sample and out-of-sample testing, you can develop a trading strategy that is robust and has a high likelihood of performing well in the future.

When moving from in-sample testing to out-of-sample testing, we want to see that the performance statistics generated across both remain relatively stable.

Summary

In summary, in-sample and out-of-sample testing are two methods used to evaluate the performance robustness of a trading strategy.

In-sample testing is used to evaluate the performance of a strategy on a set of historical data that was used to develop and optimise the strategy. Out-of-sample testing is used to evaluate the performance of a strategy on a separate set of data that was not used during the development and optimisation process.

If the performance statistics and characteristics of the strategy remain consistent throughout this process, you may be on to a winning strategy.