Why Useful Forecasting Methods Need to Be Both Effective and Efficient
Lead-time forecasts with predetermined time horizons for demand planning are widely used by supply chain practitioners for inventory planning, logistics, procurement, budget forecasting, and other S&OP applications. Once the baseline forecast has been put together, planners and managers continue to contribute their expertise in creating more useful planning forecasts with judgmental overrides or by incorporating other current data for S&OP goals and planning objectives.
It is not practical to try to find a best forecasting model, since all models are known to be wrong. Rather, we can identify forecasting Methods that are useful for a particular application.?
Regard models as useful when they are both (1) effective (doing the right things fulfilling the requirements of a particular application), and (2) efficient (doing things right in converting inputs to outputs).
We can measure the performance of these models with an effectiveness score and an associated efficiency rating. Here I propose an objective procedure that can be readily implemented with spreadsheet calculations and predictive visualizations.
What is a Profile Analysis?
A profile analysis of a forecasting Method is derived using well-established, information-theoretical concepts that are widely applied in climatology and machine learning applications. In an information-theoretical approach , a forecast error can be defined as the miss or ‘difference’ between the profiles of the actuals and the forecasts for a prescribed lead-time evaluation horizon or time period.?A Forecast Profile Error (FPE) is given by
where the ai?are the components of an Actual Alphabet Profile (AAP) and fi are the components of the Forecast Alphabet Profile (FAP). In the formula, ln stands for the natural logarithm. The units of FPE are called nats, like bits when using logarithms to the base 2.
To create a profile, individual forecasts and holdout sample actuals are ‘encoded’ or transformed into corresponding FAP (Forecast Alphabet Profile) and AAP (Actual Alphabet Profile) without changing the underlying data pattern of the forecasts and actuals. This is done by dividing a Lead-time Total into each component of the respective profiles. Thus, for a given forecast, a Forecast Alphabet Profile (FAP) is created by dividing each forecast value by the sum of the forecasts over a fixed horizon m. Likewise, the Actual Alphabet Profile (AAP) is obtained by dividing each actual by the sum of the actuals over the predetermined time horizon.
These terms I have used in several previous articles before on my LinkedIn Profile and the Delphus website Blog . The sum of the FPE over a lead-time of m periods is called the Profile Miss (FAP Miss) and can be interpreted as a measure of ignorance about the forecast profile error:
The accuracy of a FAP is given by the Profile Accuracy measure D(a|f), which is a Kullback-Leibler divergence. D(a|f) can be shown to be always positive and equal to zero if and only if forecast and actual profiles are identical. That is, AAP = FAP:
When D(a|f) = 0, the alphabet profiles overlap, which represents 100% accuracy.?The D(a|f) accuracy measure is greater than zero and equals zero if and only if ai = fi, for all i.?In other words, when the forecast pattern is identical to the pattern of the actuals, then D(a|f) = 0. Thus, the closer to zero the better.
How to Improve the Performance of a Forecasting Method
In this spreadsheet example, the twelve months starting September, 2016 were used as a lead-time holdout sample or training dataset for forecasting with three forecasting Methods: (1) the previous year’s twelve-month actuals (Year-1 ) as a judgmental Method, (2) a trend/seasonal exponential smoothing model ETS (A, A, M) , as a statistical Method and (3) a twelve-month moving average Method (MAVG-12 ) as a Naive Method.
The twelve-month lead-time forecasts make up a Forecast Profile (FP). The results show that the three MAPE s are about the same around 50%, not great but probably typical, especially at a SKU-location level.?I have also added the Naive-1 Method, because I want to use it in the effectiveness analysis below.
For the four Methods, in this example, it appears that ETS (A, A, M) is the more accurate, because the MAPE is the smallest, in this case. On the other hand, the D(a|f) profile accuracy measure makes a clearer distinction among accuracy measurements for the three Methods: MAVG-12 (= 0.096), YEAR-1 (= 0.083), and ETS (A, A, M) = 0.044.
However, ETS (A, A, M) is also the most effective, based on a Skill Score evaluation. The MAPE Skill Score yields the following results (in the brackets) where we use the Naive-1 as benchmark Method:?MAVG-12 (1- [0.54/0.50] = - 0.08); YEAR-1 (1- [0.56/0.50] = 0.12), and ETS (A, A, M) actually has the only positive MAPE Skill Score (1- [0.48/0.50] = + 0.04). It is the only Method of the three showing a positive (effective) contribution or benefit compared to the Naive-1 benchmark Method in this case.
How to Calculate the Efficiency Frontier of a Method
As shown in the May 7, 2021 and?May 21, 2021 LinkedIn articles on my Profile, the D(a|f) accuracy defines the effectiveness of a Method through the L-Skill Score for Profile forecasts.?In order to create an efficiency rating for a Method, we turn to a directional statistic based on the Profile Miss and Profile Relative Skill.
A Method is called Useful, if it is both Effective and Efficient.
领英推荐
Step 1: From the Profile Accuracy D(a|f) and Profile Miss formulae we can derive a decomposition into (1) a Profile Miss and (2) a Profile Relative Skill, as follows:
Profile Accuracy = Profile Miss + Profile Relative Skill
Because D(a|f) > 0, I can write 1 = [Profile Miss/D(a|f)] + [Profile Relative Skill/D(a|f)].?This decomposition suggests the Pythagorean theorem for a right triangle, in which Profile Miss/ D(a|f) and Profile Relative Skill/ D(a|f) can define coordinates on a unit circle with the hypotenuse as the radius. Thus, in short, the Miss/Accuracy and Relative Skill/Accuracy can be viewed as the square of a pair of directional statistics.
Once we define a reference direction, each time series represents a direction and a resulting point on the unit circle. Thus, a collection of time series forecasted with a Method creates a distribution (the Efficiency Frontier) on the unit circle.?Here are a couple of examples using the M3-competition data .
When a Method is used for forecasting many SKUs or demand items, a point of central tendency of the distribution on the circle is a measure of a typical or ‘average efficiency’ for the Method. Because we have to account for positive and negative directions, it means w need to make note of the orientation of the triangles when calculating square roots of the sides of the right triangles. (For those interested in the details, can contact me via email [email protected] for an Excel spreadsheet with this example.)
When aiming darts (forecasting) at a dart board, a forecaster is more efficient if the darts land near each other in the same direction (left target)?from the bulls-eye than if scattered around the bulls-eye in different directions (right target).?
Step 2: Using x=0, y=1 as the reference direction on the unit circle in the X-Y plane, the ‘average efficiency’ is represented by an angle on the unit circle relative to the reference direction, 0 < angle (series) < 2π.
To visualize this, if we do not hit the bulls-eye but rather an off-center ‘bulls-eye’, we can have a highly efficient Method (left target).?If we aim and strike the target at similar distances from the bullseye, but in different directions, then we have an inefficient Method (right target).
Step 3: The spreadsheet summarizes the calculations needed for displaying an Efficiency Frontier. First, the effective series are ranked with the L-Skill Score. The spreadsheet shows the results for the top ten and the bottom ten series out of a total of the 1428 monthly M3 Competition data.
Takeaway
Hans Levenbach, PhD is Owner/CEO of Delphus, Inc and Executive Director,?CPDF Professional Development Training and Certification Programs .
Dr. Hans is the author of a forecasting book (Change&Chance Embraced ) recently updated with the new LZI method for intermittent demand forecasting in the Supply Chain.
With endorsement from the International Institute of Forecasters (IIF), he created CPDF , the first IIF certification curriculum for the professional development of demand forecasters. and has conducted numerous, hands-on?Professional Development Workshops ?for Demand Planners and Operations Managers in multi-national supply chain companies worldwide.
The 2021 CPDF Workshop manual is available for self-study, online workshops, or in-house professional development courses.
Hans is a Fellow, Past President and former Treasurer, and member of the Board of Directors of the?International Institute of Forecasters .
He is Owner/Manager of these LinkedIn groups: (1)?Demand Forecaster Training and Certification, Blended Learning, Predictive Visualization , and (2)?New Product Forecasting and Innovation Planning, Cognitive Modeling, Predictive Visualization .
I invite you to join these groups and share your thoughts and practical experiences with demand data quality and demand forecasting performance in the supply chain. Feel free to send me the details of your findings, including the underlying data without identifying proprietary descriptions. If possible, I will attempt an independent analysis and see if we can collaborate on something that will be beneficial to everyone.