The Futility of Mapping Forecast Error to Business Value

A number of years ago my mom asked me if I held it against her that she beat me so much as a child. My honest answer: "No, I do not hold it against you that you beat me. I hold it against you that you knew it wasn't working, yet you kept doing it, and never tried a different approach."

I feel the same way about forecasting experts still recommending using this or that forecast error metric to quantify business value of a forecast. After many decades of trying to prove the business value of a forecast based on forecast error and never getting a satisfactory answer, experts and practitioners alike, think the solution is to try again and again, harder and harder. Einstein had something to say about that.

"Insanity is doing the same thing over and over, and expecting different results" - Albert Einstein.

Very few question the writing on the wall. Very few notice the flashing neon sign. They all bicker about what metric to use, and then offer yet another metric based on forecast error. MAPE, WAPE, WMAPE, WMdAPE, MAE, MSE, RMSE, the list goes on and on. All suffering from the same fundamental flaw: they are all based on forecast error.

Many base this on measured results in projects where dramatic changes occurred by implementing a formal forecasting solution in companies where forecast accuracy was dismal before. They fail to realize that if you change from chaos to control, ANY forecast metric will show improvement. Even the very bad ones. If you are going from 40% to 80% accuracy, any remotely related metric will get better. But that does not mean that they can measure the impact of going from 80% to 81%. And they certainly cannot help you get to 100% accuracy. Einstein had more to say about that.

“The thinking that got us to where we are is not the thinking that will get us to where we want to be.” - Albert Einstein

Then there are some people who do acknowledge the neon sign, but draw the wrong conclusion. They have come to believe that forecast accuracy cannot be mapped to business value. This however could not be further from the truth. They go on advocating to everyone to ignore forecast accuracy. A great example of throwing out the baby with the bathwater.

These experts are at least not insane. They have the clarity of mind to realize they need to stop doing what isn't working. But they have been deceived into thinking forecast error equals forecast inaccuracy, typically by their teachers and mentors of the previous generation, who still believed that forecast error was the solution.

Here is the key realization:

Forecast Error DOES NOT EQUAL Forecast inaccuracy

The fallacy that they are equal is promoted by the same experts who have built their entire careers on point-forecasting. Point-forecasting software. Point-forecasting consulting. Point-forecasting education. Point-forecasting certification. These dinosaurs are trying to hang on until retirement re-performing the same old dog-and-pony show, being recognized experts in their field. They do not want to learn a new paradigm and become a novice like everybody else. They may be so entrenched in their thinking that they may even be incapable of grasping a new paradigm that contradicts what they were taught.

Please allow me to self-plagiarize:

A point-forecaster explaining accuracy is like a color-blind person explaining color

A color-blind person can try to imagine what color is, and repeat what others have said about color, but they really do not understand the experience. Because their measurement tool, the retina in their eyes is missing photopigments. Point-forecasters, similarly, cannot experience accuracy, only repeat what others have said. Because their measurement tool, forecast error, is missing uncertainty.

The closest thing to accuracy, point-forecasters can measure is bias. Bias is a measure of the accuracy of the center of a forecast distribution, i.e. the point. It says nothing about the range of possible future outcomes. Bias is to forecasting what shades of gray are to a color-blind person. It says nothing about color. Undeniably, bias is a very valuable metric. Even when having access to true forecast accuracy metrics. But for a point-forecaster, it is truly the only useful metric they commonly use to judge value of a forecast.

As I have explained in "Your Forecast is Already Probabilistic", it doesn't have to be this way. Once you realize your point-forecast has two outputs (the forecast point and a measure of dispersion of the error of the point), and that it is nothing but a naive probabilistic forecast, you can apply metrics that you may have believed only apply to probabilistic forecasts. Metrics that actually map strongly and intuitively to business value.

These metrics come in two types, accuracy and precision, that measure complementary properties of a forecast:

Accuracy measures how far our forecast is from actual demand, whilst precision measures how narrow our forecast is. If you apply these concepts to a point-forecast, accuracy could measure bias, whilst precision could be the standard deviation of the forecast error residuals. These are the shades of gray for a color-blind person. And if the improvement made in a forecasting implementation is dramatic enough, these will indeed show a positive correlation, just like a color blind person can still distinguish night from day. But they miss the finer detail, and they miss the greatest contributors to business value erosion altogether.

In "How to Distinguish Accuracy and Precision in Forecasting" I explain in great detail how these concepts apply to demand forecasting. I strongly recommend anyone who wishes to understand how forecasts can drive business value to read that article as a pre-requisite.

Demand forecasts output two things: the central point and the distribution around the point. For probabilistic forecasts the distribution of demand uncertainty is the main output, and a central point of choice, typically median or mean of the distribution, can also be provided for use in deterministic planning processes. For statistical forecasts the point is the main output, and the distribution of forecast error residuals is a secondary output. This secondary output does typically get used. Most often to calculate safety stocks: the point-forecasters lip service to uncertainty. Importantly:

Both outputs get used in business. Point and distribution.

And equally important:

Both outputs impact the business. But differently.

The point drives business-as-usual. It is the baseline of all our plans, and thus crucially important. And when the point is sufficiently wrong, it can hurt the business. This happens for example when we launch a new product or run a promotion and we think the product will be an enormous success but the market isn't excited. We stock millions and sell thousands. In such extreme cases the accuracy of the point will indeed matter a lot. But for everyday demand and supply behaviors the point is meaningless beyond being the baseline.

The distribution is a different story. In everyday supply chain behaviors the average value has no impact. It is at the extremes of our demand distribution where we feel the impact on the business. If demand comes in much higher than expected we may stockout, hurt customer service, and lose sales. If demand comes in much lower than expected we are stuck with unsold goods consuming warehouse space and working capital, or if our product expires it becomes waste.

The accuracy of the demand uncertainty distribution is the driving factor behind these disruptions. If the distribution is wrong, especially the tails, it erodes margin. The precision of the demand uncertainty distribution provides the safety buffers. Since precision is known at the time of forecasting, it is the driving factor of buffers like safety stocks

Precision drives business efficiency

The narrower we can make our uncertainty distributions, i.e. the greater our precision, the more efficient our buffers become in protecting against uncertainty. We can measure this impact on the business easily. But when our uncertainty distribution is incorrect the buffer breaks and we get hit with disruption. We need to measure the accuracy of the distribution, and again especially the tails of the distribution, to capture this.

Accuracy drives business stability

Any inaccuracy will cause margin erosion. If you want to measure impact on the business, margin erosion is the dominant factor. Only accuracy of the distribution captures this. There are metrics that can do so properly (e.g. Total Percentile Error). Between forecast error and these two facets of the forecast distribution the following should be intuitively clear:

The point forecast error captures the occasional swoop of the broadsword. The forecast distribution captures the continuous death by a thousand paper cuts.

We cannot ignore the big swoops. We must measure bias. Especially before and after making big assumptions on promotions, product launches, and other major non-everyday decisions. But what hurts businesses continously is the margin erosion caused by inaccuracy. If we want to measure the impact of the forecast on the business, we cannot ignore this dominant factor. We must measure accuracy!

The key epiphany needed is that:

Forecast error cannot measure this. It is fundamentally impossible!

Forecast error only measures the point. Forecast error ignores the extremes of the demand uncertainty distribution. It ignores the very thing that causes the impact on the business. It is impossible for forecast error to map to business value since it ignores the impacts of business value...

The only metrics based on forecast error that are useful are bias and relative bias. Every other metric, regardless whether it takes absolute values or squared values of forecast errors, has no additional value. These are just very poor stand-ins for accuracy and precision that fail in practice because they fail to measure that which impacts the business.

Anyone who claims some forecast error metric of their choice is better at measuring business impact than some other forecast error metric hasn't had the epiphany yet that if it ignores the impact it cannot measure the impact. The futility of this proposition should be evident. Believe such experts at your own peril...

One final note.

Use the right metric for its purpose

There is nothing fundamentally wrong with many forecast error metrics. WMAPE is useful. MAE is useful. But they are useful for troubleshooting. If you aim to find which items have the largest issues, by all means use these metrics. They work better than accuracy metrics for this purpose. But if you want to determine value of a forecast absolutely never ever use these. They are broken compasses. Use true accuracy and precision metrics instead.

And if you want to determine Forecast Value-Add (FVA) do not simply use WMAPE or other forecast error metric as the underlying metric. Your FVA will be as broken as those metrics are. Use a true accuracy metric as the basis of your FVA. You can find a clear explanation on how to use FVA for best business impact in "Perspective on Measuring Forecast Value-Add".

Every purpose has a metric or set of metrics that serve it. If your purpose is to measure the impact a forecast has on your business, bias, accuracy, and precision will serve you well. Accuracy and precision are necessarily probabilistic metrics: they measure the distribution, not the point. If a point-forecaster tries to tell you some forecast error metric of choice is an accuracy metric, just remember they are a color-blind person trying to explain color...

Now that I've caught your attention, in a next article I'll dive deeper into the shades of gray, and nuance how traditional error metrics may still capture a small part of the impacts.

If you are interested in probabilistic metrics and probabilistic planning and forecasting please consider joining the "Probabilistic Supply Chain Planning" group here on LinkedIn.

Find all my articles by category here. Also listing outstanding articles by other authors.