Time series models are useful for analyzing and forecasting data that change over time, such as sales, weather, or stock prices. However, how do you know if your model is accurate and reliable? In this article, you will learn how to validate a time series model using various methods and metrics.
The first step to validate a time series model is to split the data into two parts: a training set and a test set. The training set is used to fit the model, while the test set is used to evaluate its performance. The split should preserve the temporal order and structure of the data, and the test set should be large enough to capture the variability and trends of the time series. A common rule of thumb is to use 80% of the data for training and 20% for testing.
Using domain knowledge in splitting time series data is the recipe for success. Understand the trend, business cycle, internal/external event (burn incentives, economy), stationary vs non-stationary, and frequencies matter to know which data to use. Using Walk Forward validation is usually very useful with the downside is computation heavy. Understand the concept drift. If you still not sure, its always better to clustering and make split of data in timeseries.
The second step to validate a time series model is to check the residuals, which are the differences between the observed and predicted values. The residuals should be random, independent, and normally distributed, with zero mean and constant variance. You can use various plots and tests to check these assumptions, such as histograms, autocorrelation functions, Ljung-Box tests, and Q-Q plots. If the residuals show any patterns, correlations, or outliers, it means that the model is not capturing some features of the data and needs to be improved.
Residuals should be independent, and no autocorrelations. This are fundamental assumption in time series data. If the residuals are autocorrelated, then something wrong by the applied model (under-fitting, etc). In general we assume variance of residual is constant (homoscedastic). When its become heterescedasticity, change of variable will make the variance of the residual change as well. There are statistical and visual approaches to check on this like Ljung-Box test, Q-Q plots or ACF.
The third step to validate a time series model is to compare the forecasts with the actual values in the test set. You can use various metrics to measure the accuracy and error of the forecasts, such as mean absolute error (MAE), root mean squared error (RMSE), mean absolute percentage error (MAPE), and R-squared. You can also use graphical methods, such as time plots and error plots, to visualize the performance of the model. You should choose the metrics and plots that are most relevant and meaningful for your problem and data.
Validate time series is very tricky and can be back-fired. MAE, RMSE, MAPE, SMAPE are primarily measured on error, not direction, also its have problem with dealing with outliers, over/under-estimation, zero/near-zero number. Its must be consulted with a business nature about the tolerance on under/over-estimation and sensitivity to outliers. For instance, if there is spike event over 2 days duration with big error gap, is that acceptable or not. Several way like Diebold-Mariano Test and Volatility Measures can be alternative solution
The fourth step to validate a time series model is to perform cross-validation, which is a technique that uses multiple splits of the data to estimate the generalization error of the model. Cross-validation can help you avoid overfitting or underfitting the model, and select the optimal parameters and features. There are different types of cross-validation methods for time series data, such as rolling window, expanding window, and k-fold. You should choose the method that best suits your data size, frequency, and seasonality.
The fifth step to validate a time series model is to test its robustness, which is its ability to handle changes and uncertainties in the data. You can test the robustness of your model by using different scenarios, such as adding noise, outliers, or missing values to the data, or changing the time period, frequency, or scale of the data. You can then compare the forecasts of your model under different scenarios and see how they differ from the baseline. You should aim for a model that is stable and consistent across different scenarios.
The sixth and final step to validate a time series model is to benchmark it against other models or methods. You can use simple or complex models, such as naive, exponential smoothing, ARIMA, or neural networks, and compare their forecasts and errors with your model. You can also use external or historical data, such as industry averages, expert opinions, or previous forecasts, and compare them with your model. You should choose the models or methods that are relevant and comparable for your problem and data.