A real-world application of Ensemble Methods in time-series forecasting

Recently, I had the opportunity to work with one of the world’s largest oil and gas service companies. This company provides the industry’s most comprehensive selection of advanced technologies ranging from reservoir characterization to drilling and production. The project with which I was involved covered all three levels of analytics applications: descriptive, predictive, and prescriptive.

During the predictive analytics phase, we discovered that applying Ensemble Methods to Time Series Forecasting produced exceptional prediction performance. As a result, we were able to use our model predictions to develop highly optimized prescriptive analytics that saved millions of dollars in operating costs. Next, we will discuss the approaches used for predictive analysis in a real-world scenario.

The Baseline Model

The aforementioned company needed to predict asset demand to manage short-term asset inventories and maintenance schedules. They had been using a simple, effective model to estimate the asset demand forecast. For each asset, this model applied the moving average of 90 days’ daily demand as the 30 days future demand forecast. Our predictive analytics objective was to outperform the existing baseline model through various time-series data modeling techniques.

To determine the most effective model, we planned to try several modeling techniques on a given training dataset by evaluating the fit and generalization of each technique on both the target variable, which was asset demand in this case, and on the test dataset. Lastly, we evaluated how well they generalized by applying the Monte-Carlo method on the validation dataset.

Modeling Approach

In our modeling approach, we applied three predictive analytics techniques for demand forecasting: Univariate Time-Series Analysis, Multivariate Forecasting, and Multivariate Ensemble Forecasting.

1. Univariate Time-Series Analysis

To predict the future demand D = {D1, D2, … D30} at the future time from the next day to future 30 days, the predictive model can be expressed as a function of past demands H = {H-90, H-89, … H1} over a historical time window from the past 90 days till today, or d = f(x); x ∈ H, d ∈ D. We need to take into consideration that data points closer to the current moment have different weights than data points further back in history, and that certain intrinsic patterns within a time window can repeat in the future. 

By applying moving average decomposition to the demand time series data, we were able to identify the effects of key time series components such as seasonality, trends, and irregularities. This offers a better understanding of the nature of historical demand data and the performance of the baseline model based on a moving average. The figure below illustrates the decomposed time series of asset demand for each asset type.

As shown by the “random” component from the charts above, the results of decomposition are not truly random, which means the simple moving average baseline model does not capture all factors contributing to the demand prediction. In effect, there are likely other opportunities for better forecasting with more advanced modeling techniques.

The available datasets were analyzed using the following univariate models:

• Holt-Winters
• Auto ARIMA (Autoregressive Integrated Moving Average Model)
• Auto ARIMA with Fourier series decomposition

1.1 Holt-Winters

Holt-Winters is an Exponential Smoothing algorithm used to analyze univariate time series data that exhibit both trends and seasonality. Compared to the simple moving average analysis, it uses exponentially weighted moving averages to model data[1].

This figure shows the Holt-Winters model’s 30-day demand predictions for all three assets, along with the 80% and 90% confidence intervals which are highlighted in red. The actual demands are plotted for comparison as well.

1.2 Auto ARIMA

The Autoregressive Integrated Moving Average model (ARIMA) is another widely-used approach to time series forecasting. ARIMA models aim to describe the autocorrelations in the data that are non-stationary, as opposed to the exponential smoothing model which is based on the description of trends and seasonality in the data.

The figure here shows the Auto ARIMA model 30-days demand predictions for all three assets.

1.3 Auto ARIMA with Fourier Series Decomposition

A different route to analyze the time-series data is to model the long-term effects of multiple seasonalities. Fourier series decomposition is the natural next step to determining if the model accuracy would be improved.

Fourier series decomposition is widely used in engineering to decompose signals into different frequencies. A similar concept was used to analyze multiple seasonality based on the Auto ARIMA model. The number of Fourier terms was selected by minimizing the AIC[2]. However, our findings show that the model performance was less effective than the Auto ARIMA model due to the increased complexity of that model. This figure shows the 30-day demand predictions for all three assets using Auto ARIMA with Fourier series decomposition model.

There is beauty in simplicity when it comes to solution design, and not surprisingly, simplicity also works better in mathematical modeling. We achieved better results by applying the principle of Occam's razor in data modeling. Because all assumptions introduce possibilities for error; if an assumption does not improve the performance of our model, its only effect is to increase the probability of introducing biases.

To compare the performance of different models, we ran multiple simulations of prediction forecasts. Then, for each simulation, we calculated the root mean square error (RMSE) of the prediction error against the actual demand for each day. As shown in the figure below, the ARIMA model has the smallest RMSE median. For Assets A and C, the range of ARIMA boxplot quartiles is also much smaller, meaning that the ARIMA model is a more generalized and stable model. 

When comparing the three univariate models, the Auto ARIMA outperformed both the Holt-Winters and Auto ARIMA with Fourier decomposition.

2. Multivariate Forecasting 

Multivariate forecast models allow us to consider both econometric data and historical demand data, unlike the univariate time series analysis which models only the volatility of demand.

In this case, the average demand over the future 30 days was our target variable, and the predicted average demand was our daily demand forecast for all future 30 days. Here, we define the target variable of future demand as D for the future month. We can express the model as a function of various econometric factors and historical demands as aggregated over a fixed time window. For our purposes, at the month n, we defined Xn = {X1, X2, …} as a list of proprietary econometric data for the month, and H = {H(n-1), H-(n-2), …} as the past monthly demands. We aggregated the data monthly and defined the forecast function as D = f(x, g(h)); x ∈ Xn, h ∈ H.

We then analyzed the given data set by using supervised learning models, such as support-vector machines (SVM), random forests, and multivariate adaptive regression splines (MARS).

2.1 Support-Vector Machines

The SVM model searches the optimal hyperplane with the maximum margin based on the training data. Here, we used the SVM model to classify predicted demand to ranges of possible values. The figure below applies the SVM model to show the predicted average demands versus the actual average monthly demands for all three assets.

2.2 Random Forests

Random Forest is a supervised learning model based on the decision tree. Typically, a simple decision tree model tends to overfit training data sets. The Random Forest algorithm corrects this issue by constructing a multitude of decision trees and applying ensemble methods to obtain better predictive performance. The figures below use the Random Forest model to compare the predicted average demands versus the actual average monthly demands for all three assets.

2.3 MARS (Multivariate Adaptive Regression Splines)

MARS is a non-parametric regression technique that can be considered an extension of recursive partitioning approaches to regressive models. It provides a flexible way to model high dimension data while capturing nonlinearities and interactions between variables[3]. Below, the MARS model shows the predicted average demands for all three assets compared to the actual average monthly demands.

Using the same evaluation criteria as the univariate approach, we treated the predicted average demand as the predicted demands for the next 30 days. We calculated the distribution of RMSE by comparing the actual demand of each day with the predicted average demand. We used the “performance estimation” package in R Programming Language to evaluate multiple multivariate models simultaneously. The following figure demonstrates the demand prediction RMSE output.

In general, multivariate models outperform univariate time series models. For Assets A and C, MARS was the best multivariate model, while SVM delivered the best results for Asset B.

3. Multivariate Ensemble Forecasting

Ensemble is a machine learning modeling technique that trains multiple models on the same data set, then combines them to produce better results than individual models. The diversity of the individual models drives a high-performance for the ensemble model. In this context, diversity implies that all the individual models perform well, but in different situations. Inspired by this concept, we attempted to combine the power of econometric multivariate modeling and the concept of univariate modeling. The ensemble then captured the causal relationship between econometric and asset demand in addition to the dynamic weighting of time series data at different points

We applied a unique Ensemble Modeling approach that does not require a commitment to a single fixed time window in the historical data for model training[4]. For example, a 90 day fixed time window can be applied to the multivariate models, while a 180 day fixed time window can be used for the univariate model.

Our ensemble approach first trains several multivariate models. Each multivariate model aggregates data using different time windows, then the ensemble model produces the final prediction by aggregating predictions from each of the multivariate models using the bagging ensemble method. In our case, we trained monthly, quarterly, biannual, and annual models. We expected that each of these aggregations of training data would capture the short-, mid-, and long-term effects of intrinsic patterns embedded in the time series. We engineered the data features of these models so they would exhibit the diversity of prediction results needed to produce an improved ensemble model.

As with the multivariate model, we used the average demand over the future 30 days as our target variable and used the predicted average demand as our daily demand forecast for all future 30 days. At time t, in order to predict the future demand at the future time t+1, we defined the average demand prediction results of each multivariate forecasting model as f1(t+1), f2(t+1), … etc., and we defined the ensemble future average demand prediction as X = {f1(t+1), f2(t+1)...} at time t+1. The ensemble forecast model can be expressed as the function of D = f-ensemble(x); x ∈ X. The figure below shows the concept of combining multiple models using ensemble.

3.1 Feature Engineering

In an effort to capture the short-, mid-, and long-term effects of intrinsic patterns embedded in the time series, we engineered more sophisticated data features when training individual multivariate forecasting models. In the short-term monthly model, we added the maximum and minimum daily demand, the daily demand moving average, and the daily demand standard deviation to model the volatility of demand in the short term, in addition to the econometric data. For the mid- to long-term models, we added the moving average and standard deviation for econometric data features to model the effects of trends.

Below are the snapshots of demand forecasting comparisons between the baseline model and the trained model in one of the simulations. All data points for the future 30 days were held out to the prediction models. As a reference, the true average future 30 days demand is plotted in green.

These figures indicate that the ensemble forecast is closest to the true future 30 days average demand. We observe this consistently throughout the model evaluations using the Monte-Carlo simulations, which will be discussed in greater detail in the next section. The ensemble approach provides a powerful model that simultaneously captures the predictive factors while extracting patterns of time series in various historical time windows.

4. Model Evaluation

During the training of various models, we held out two years of actual data to evaluate the performance of these models. We compared the error between the predicted and actual demand on the selected test datasets with Root Mean Square Error (RMSE), one of the most common measures in predictive analytics.

Time series data have a natural temporal ordering; therefore, a simple, random sampling technique is inappropriate for time series predictive models. To properly evaluate our models, we combined the “sliding window” test strategy with the Monte-Carlo simulation.

For a single simulation, we randomly sampled a time point in the available holdout data time range. We then took a fixed sliding window of all the past data up to the sampled time point as the input to the trained models. The models predicted the demand for the future 30 days, at which point we calculated the RMSE between the actual 30-day demand and the predicted demand. In a trial run, we ran 3000 simulations for each trained model and obtained a distribution of RMSE. This figure illustrates the evaluation approach:

As previously mentioned, the company used a 90-day moving average as the model for the future 30-day average demand forecast. We used this as our baseline model for prediction performance comparison. For each simulation, we needed to show the predictive ability of our models. To do this, we determined the future 30-day average actual demands that our models were trying to forecast and labelled them as the “perfect model”. We then calculated the corresponding RMSE value between the perfect model and the daily demand. Understanding the 30-day average demand as if we knew it precisely is the best possible model RMSE.

The RMSE value of the future 30-day average demand is representing the daily volatility, which is a natural phenomenon. We have also found that the difference in daily volatility was close to a normal distribution. In a future article, I will discuss how we used an innovative method to manage asset planning using Prescriptive Analytics according to the normal distribution in daily volatility.

Based on the distributions of RMSE from simulation results, a successful model should outperform the baseline model. The best model’s RMSE should perform as close to the future 30-day average demand as possible.

Below, we compare the RMSE distributions of the baseline model, our improved models, and the perfect model of the future 30-day demand average.

Clearly, the ensemble model outperforms all other modeling techniques and produces exceptional prediction performance closest to the true future 30-day average.

Our project demonstrated how analytics could create impacts on business process improvements and achieve a number of strategic goals in a highly scalable manner. I would like to thank @Alexander de León Casta?o, @Ashwin Ramani, @Abdulaziz Alsalim and @Modhar Khan. I hope this article inspires more initiatives in applying ensemble modeling to time series data.

[1] P. S. Kalekar, "Time series forecasting using Holt-Winters exponential smoothing," Kanwal Rekhi School of Information Technology, 2004. https://labs.omniti.com/people/jesus/papers/holtwinters.pdf

[2] Hyndman, Rob J. 2010 Sep 29. Forecasting with long seasonal periods [blog]. Hyndsight. https://robjhyndman.com/hyndsight/longseasonality/

[3] Friedman, Jerome H. Multivariate Adaptive Regression Splines. Ann. Statist. 19 (1991), no. 1, 1--67. doi:10.1214/aos/1176347963. https://projecteuclid.org/download/pdf_1/euclid.aos/1176347963

[4] Oliveira, M., Torgo, L.: Ensembles for time series forecasting. In: Proceedings of the Sixth Asian Conference on Machine Learning, pp. 360–370 (2015). Also available at: https://proceedings.mlr.press/v39/oliveira14.pdf