Forecasting when accuracy is no longer the goal: a new world of possibilities.

Business forecasting requires business-oriented metrics, which accuracy metrics are not. Bridging this gap unlocks multiple use cases companies and practitioners have long been waiting?for…

In this series of articles[1], we’ve so far demonstrated why and how existing forecast accuracy metrics prevent demand planners from delivering more business value.

More precisely, using data from the M5 competition, we’ve empirically established the weak correlation between accuracy metrics and cost-effectiveness.?

In other words: existing metrics do not take into account the intended use of a forecast and the associated costs… Under such conditions, minimizing costs (or maximizing profitability) through improved forecasting is more a matter of luck than science!

How frustrating for companies and practitioners to waste time and resources working hard only to find that it does not always add the expected value (and worse, often reduces profitability)!?

But the good news is that any demand planner could start creating more business value by simply switching from “accuracy” to “costs” metrics.

Focusing on “costs” opens up a whole new world of possibilities for demand planning and allows for half a dozen exclusive use cases!?

In this article, I will illustrate some of them. But first, let’s take a quick look at what “Decision Impact” metrics are.

What are “Decision Impact”?metrics?

The very first article of this series[2] reminded us that forecasts are not an end in themselves, but serve to support decision-makers so that they can make better decisions.

The purpose of forecasting is not and has never been to provide the best forecast ever! Its purpose is to enable the best decision.

The best forecast is therefore not the perfect forecast, but the one that allows the best decisions to be made. The mission of forecasters should therefore not be to minimise the error between a forecast and reality but to minimise decision errors.

This means moving from measure “Forecast Accuracy” (FA), which focuses on the intrinsic quality of the forecast produced, to measuring “Decision Impact” (DI) which focuses on the final forecast-based decisions and their relevance/cost and impact.

A generic cost?function

The proposed metrics require the creation of a “cost function” that can, for any forecast input, evaluate the business decision and assess its quality.

Although the quality of a decision can be expressed in many ways, in this study we have opted for financial costs.

The creation of such a “cost function” was also detailed in the first article of this series[2]. Let us assume here that such a “cost function” is defined. It then allows us to calculate the cost associated with any input forecast.

Cost metric #1: Actual cost?(DIa)

By applying the cost function to our “actual” forecast, we obtain the cost induced by our forecast. Let’s denote it by DIa.?

Cost metric #2: Naive cost?(DIn)

By applying the cost function to a “naive” forecast (for example, a moving average forecast), we obtain the cost induced by the simplest forecasting process we could implement. Let’s denote it by DIn.

Cost metric #3: Oracle cost?(DIo)

By applying the cost function to the “oracle” forecast (ie. the actual sales), we obtain a posteriori the cost that perfect knowledge of the future would have induced. Let’s denote it by DIo.

Combining the DIa/DIn/DIo building?blocks

These three costs are elementary building blocks that can easily be assembled to create three insightful metrics.

Measuring the earned?value

What is the performance of the forecasting process? Is it generating value? How much?

When it comes to measuring their added value to the company, demand planners often fall short. As their key role usually ends with the delivery of a qualitative forecast, their only asset is the FA. They have little ways to evaluate business value.

Yet there are attempts to assess their performance.?

One of them is to compare FA to industry benchmarks. But Nicolas Vandeput recently reminded us in a recent article[3], there are many reasons why benchmarks should be avoided at all costs!?

These reasons include the diversity of business strategies (size of portfolio, product & brand positioning), the level at which the forecast accuracy is measured and even the metric’s definition itself (especially with value-weighted formulas) may differ, etc.

Another approach is to apply FVA (Forecast Value Added). Most of the time, it consists in comparing the actual FA against the FA of a naive forecast to assess whether the forecast has been improved or not.

This sounds great, and it is! But there are still important limitations to the way it is generally applied:

• FA is not a key business performance indicator (KPI). As we have said, FA has little correlation with business performance. So improving FA does not mean you are generating value. Costs may increase, decrease or remain the same as accuracy changes.?
• FA metrics contradict each other. There are dozens of FA metrics and they’re not all equal. Worse still, they often contradict each other: one improves while the other deteriorates. Thus, a simple switch from one FA metric to another could profoundly alter your FVA results.
• FVA applied to FA is not FVA, but FAA. As Paul Goodwin summarised in a recent chat, “Forecast Value Added (FVA), as it is often applied, is really Forecast Accuracy Added (FAA) and this can be misleading.” Indeed, as long as FA is used in FVA, the true value will be out of reach...

Don’t get me wrong. I’m a big fan of FVA and Mike Gilliland’s work (the godfather of FVA)[4]. His work has been a great inspiration for this series of articles.

But, as long as FA is used in FVA, FVA will have as little correlation to business value as?FA.

Interestingly, there’s no real problem with the FVA approach in itself! In a 2019 ISF presentation[5], Mike Gilliland gave this definition of FVA:?

The change in a forecasting performance metric that can be attributed to a particular step or participant in the forecasting process.

It is therefore clear that FVA can be applied to any metric. Replacing FA with DI is therefore not only respectful of the definition of FVA but also much more natural as long as evaluating business value is the objective.

Let us then calculate DIna, the difference between DIn (naive cost) and DIa (actual cost).

What does DIna measure? Nothing less than FVA: the added value of the forecasting process from a business perspective.

Measuring the yet-to-be earned?value

Should you be satisfied with your current DIna performance? Generating value is good… but have you captured it all or are you just scratching the surface?

Interestingly, according to FA metrics, the best performance you can dream of is 100% accuracy! And this is the same for every SKU. This is the Holy Grail of demand planning!

But what’s the added business value of 100% accuracy? 1 million dollars or 1 dollar? Well… once again, we are at a loss!

How can DI metrics help us?

Let’s follow the same logic as before and introduce DIao: the difference between DIa (actual cost) and DIo (oracle cost).

What does DIao measure? Nothing less than the unearned value: the value that could still be gained by improving forecasts.

For sure, it is not possible to capture 100% of the DIna value for every SKU, as it would require a perfect knowledge of the future. However, DIao provides one of the most interesting insights we can dream of… as it helps identify where the value really lies. The use cases described below will tell you more about this.

Measuring the earnable?value

So far, we have presented DIna and Diao? Let’s evaluate DIno. What would it measure?

Following the same logic as before, DIno is the difference between DIn (naive cost) and DIo (oracle cost). As such, it could also be defined as the sum of DIna (earned value) and DIao (not-yet-earned value).

DIno is therefore a measure of the full playing field: the total earnable value.

Summarizing it?all

• DIn: cost of naive forecast
• DIa: cost of the actual forecast
• DIo: cost of oracle forecast
• DIno measures the total available value
• DIna measures the earned value
• DIao measures the yet-to-be earned value
• DIna/DIno measures the proportion of the earned value
• DIao/DIno measures the proportion of yet-to-be earned value

Seven use cases finally?unlocked

Sharing forecast performance

“The forecast is always wrong!”, “The forecast is too this”, “The forecast is not enough that”, “What does 70% FA mean?”, “Is this good or bad?”

Demand planners often need to share and defend their performance in an objective, easy to understand and non-contradictory way with a very diverse audience.

So far, FA communication is the rule. But let’s be honest,?

• Most (if not all) FA metrics are difficult for non-practitioners to understand.?
• A given FA level is a fact and does not say if it is good or bad
• FA metrics contradict each other: one metric being improved while another deteriorates.

Under these conditions, it is almost impossible to communicate effectively through FA metrics. In every company, this often leaves room for scepticism not to say criticism.

DI metrics are metrics that express financial costs, gains or losses. This makes them easy to understand, no matter who reads the figures.?

With the DIna metric, demand planners can now demonstrate that, even if the forecast was not perfect, it still has saved a given amount of money. Conversely, they can accurately assess the cost of poor performance and proactively trigger the necessary work.

Establishing the ROI of forecasting

What’s the quarterly, semestrial or annual ROI of the demand planning department??

All major business decisions require knowledge of return on investment. If you don’t know whether your efforts are producing results, how can you properly maximize profitability? Yet, in demand planning, we are playing a guessing game.

But now, the DIna metric offers an insightful perspective on the added value for the company. It is then very practical to determine the ROI of forecasting.

To illustrate this, let’s look back at some of the lessons from the M5 competition [6]. If Walmart had applied the “F_ADIDA” method to generate its forecast, the DIna metric establishes that the forecast would have saved $2,345 (compared to a naive “moving average” method) over the scope of the competition (10 shops, 3049 items, 1 month). By subtracting some internal forecasting costs (people, tools, etc.), the ROI of forecasting is then easily calculated.

Promoting best practices and eliminating bad?ones

Do judgemental forecasting makes things better or worse? Does this X or Y data improve the forecasts? Which forecasting model adds the most value?

When these questions are raised, the usual answer is “FVA”. This is the right answer… as long as the added value is indeed measured. That’s precisely where DIna comes in.

If you’re not yet familiar with FVA, then I highly recommend watching this excellent webinar[7] in the “CMAF Friday Forecasting Talks” series by CMAF, Lancaster University.

Evaluating fairly forecasting staff performance

When it comes to evaluating individual performance, are you able to identify the best and worst performers?

Of course, assessing someone’s performance on business value-added alone is far too restrictive. Human beings are not numbers on paper.

However, it is also not fair to discard this information and not be able to identify the best and worst performers.

Very often, performance is assessed with FVA (accuracy based), usually weighted by portfolio revenue, volume or number of items. But this approach does not do justice to everyone work. Some perimeters are easier or harder to forecast than others, some are larger, some are more critical, etc.

But now the “Value-added” (DIna) or “Proportion of value-added” (DIna/DIno) calculated by demand planner provides a clear and factual measure of each individual’s contribution to the company, allowing a better appreciation of the work done and the specificities of each portfolio.

Building a strong business case for investment

Should we invest in an advanced solution? Does it worth it? Which vendor’s solution is best suited to my specific context?

Investing in a new demand planning solution usually requires three steps?:

• Align the management team to get the green light and initiate a project
• Identify the solution that best meets your needs
• Post-deployment, ensure the solution delivers the expected value.?

In demand planning, FA plays a central role in each of these three steps. As such, the vendor with the best FA holds the best cards!

But improving FA is not a business case. It’s therefore not the right KPI to demonstrate the need for improvement, nor choose the right solution nor assess the value it delivers.

In contrast, DI metrics are cost metrics that naturally support such an investment process. DIao demonstrates the improvement potential from a business perspective. In turn, DIna assesses the actual added value of each vendor to identify the supplier that offers the right solution for the intended use. Finally, the monitoring of DIna over time ensures that the project has delivered (and continues to deliver) the expected value in the long term.

Once again, to illustrate this, let’s look back at some of the lessons from the M5 competition [6]:?

• As stated previously, if Walmart had applied the “F_ADIDA” method to generate its forecast, the DIna metric establishes that the forecast would have saved $2,345 (compared to a naive “moving average” method) over the scope of the competition (10 shops, 3049 items, 1 month).?
• The DIao metric establishes that the improvement potential is $21,946 (on the same small perimeter). Thus ~10% of the total addressable value has been secured by the current method. Would such potential be worth triggering a project? Probably.
• To improve its performance, Walmart might be interested in applying the “YJ_STU” method that won the M5 competition. By doing so, it would have saved $2,981 (compared to a naive “moving average” method). The value added of this method would therefore be $636 compared to the current method.
• Let’s imagine that Walmart also asked for a proof of concept from another “supplier”, namely “Hiromitsu Kigure”. This method was ranked #45 in the competition, which means that based on the forecast accuracy metric applied, this method is not as good as “YJ_STU”. However, the DIao of this method reveals that Walmart could have saved $5,000 (compared to a naive “moving average” method). The value added of this method would therefore be $2,655 compared to the current method, which is more than 4 times higher.
• Of course, the project costs (personnel, tools, etc.) still need to be taken into account in order to have a fair comparison of these two suppliers and to decide which solution is best for the specific Walmart context.

Prioritizing work

Should I be satisfied with my current performance? Should I continue to improve should I stop? Where should I focus on first? How good is good performance?

Not all SKUs are created equal! Some are hard to predict, some are not. Some are high stakes, some are not. Some are strategical, some are not. Some have strong supply constraints, some are not. And so on.

Under these conditions, it’s not logical to expect the same performance, nor to invest the same energy for each SKU.

How do you define the scope to be examined in priority? Are the items with the lowest FA a priority? Lots of practitioners apply the ABC/XYZ classifications, does this help focus on the right scope?

Well, unfortunately not.?

Let’s not dwell on FA, as we’ve already discussed its value to the business.?

Regarding ABC/XYZ limitations, I recommend you read this article by Nicolas VanDePut: “ABC Analysis is Not a Good Idea. Do This Instead.”[8]. This article clearly states the limitations and offers some ideas on how to focus on the right scope. Guidelines include focusing on value, shelf-life, holding costs, supply lead times, end of season and criticality.

It is interesting to note that most (if not all) of these elements are related to the costs of the triggered decision. As such, they are properly reflected in the DI metrics.

In other words, DI metrics naturally rank items and prioritise those that have the most impact (i.e. have a high DIao value to gain).

Taking advantage of probabilistic forecasting

When moving from deterministic to probabilistic forecasts, practitioners need to replace their good old deterministic accuracy metrics with probabilistic ones.?

The first good news is that this does not apply to DI metrics! Because they focus on decisions and costs, not on the forecast itself, DI metrics are by design compatible with any type of forecast.

The second piece of good news is that using DI with probabilistic forecasts provides additional insights!?

Here is one. Probabilistic forecasts accurately describe possible future events and their probability of occurrence. This is a great way to identify perimeters at risk and prioritize decisions review.?

For example, let’s say we focus on the 5% to 95% percentile range which covers 90% of demand values. The higher the uncertainty in the forecast, the larger the range. Consequently, the width of this range could be seen as an interesting way to prioritize the review of perimeters with high uncertainty.?

That’s great. Yet, high uncertainty does not necessarily mean that business is at risk. What really matters is the economic risk of this uncertainty!

How do we measure the economic risk of uncertainty? We could leverage DI metrics for this!

Let’s denote 1) “DIa min” the costs associated with the forecast at the 5% percentile and 2) “DIa max”, the costs associated with the forecast at 95% percentile. The absolute difference between “DIa min” and “DIa max” defines the actual risk of extra costs carried by uncertainty.?

Which items should then be reviewed first? As you have already guessed, those with the highest economic risk of uncertainty represent the real risk area.

Conclusion

Business forecasting requires business-oriented metrics. This seems obvious, but what goes without saying, goes even better when it is said.

FA metrics are not business-oriented and experience shows they cannot be considered as a proxy for such metrics.

Introducing cost metrics such as the proposed DI metrics to your “hall of fame of forecasting metrics” is not as difficult as it sounds, and it opens up new perspectives and exclusive use cases!?

This article aims to shed light on current practices, limitations and possible improvements of forecast performance measures. It’s for sure not perfect and suffers from limits.

If you found this to be insightful, please share and comment… But also, feel free to challenge and criticize. Contact me if you want to discuss this further!

In all cases, stay tuned for the next articles! In the meantime, visit our website www.vekia.fr to know more about our expertise and experience in delivering high value to Supply Chain.

Linkedin: www.dhirubhai.net/in/johann-robette/

Web: www.vekia.fr

References?

[1] Vekia, J. Robette, “Decision Impact metrics” articles - Table of content , 2021

[2] Vekia, J. Robette, “Decision Impact”: 10 reasons to implement the new generation of business-oriented metrics , 2021

[3] Nicolas Vandeput, “Assessing Products’ Forecastability: Forecasting Benchmarks vs. COV” , 2021

[4] Mike Gilliland, “The Business Forecasting Deal” , 2010

[5] ISF, Mike Gilliland, “Forecasted Value Added Analysis ”, 2019

[6] Vekia, J. Robette, ““The last will be first, and the first last”… Insights from the M5-competition ”, 2021

[7] CMAF, Robert Fildes, Mike Gilliland, “What do we need to know about Forecast Value Added?” , 2020

[8] Nicolas Vandeput, “ABC Analysis is Not a Good Idea. Do This Instead.” , 2021