How do you identify and remove outliers in demand forecasting data?

Distort Forecasts: Outliers are extreme values that deviate significantly from the normal pattern of demand. They can distort statistical calculations used in forecasting models, leading to inaccurate predictions. Inflate Error Metrics: Outliers can inflate error metrics like Mean Absolute Percentage Error (MAPE) or Mean Squared Error (MSE), making your forecasts appear less accurate than they actually are. Mislead Decision-Making: Inaccurate forecasts caused by outliers can lead to poor inventory management, over/under production, and missed sales opportunities.

In the example above where a drop in demand was due to a data error the use of Bayesian Inference or Bayesian Probability can be used to smooth the demand curve. However note, discovering a drop in demand like this may be a signal that an out-of-stock has occured, or even worse inventory was at the location however it was sitting in the back warehouse and not out on the selling floor.

Visual Inspection: Time Series Plots: Plot your demand data over time and visually scan for any data points that significantly deviate from the overall trend or pattern. These could be spikes, dips, or unusual fluctuations. Scatter Plots: If you have multiple variables that might influence demand, create scatter plots to see if any data points are far away from main cluster of points Clustering Algorithms: Algorithms like DBSCAN (Density-Based Spatial Clustering of Applications with Noise) can group similar data points and identify outliers as points that don't belong to any cluster. Isolation Forests: This machine learning algorithm isolates outliers by randomly partitioning the data and measuring how easily each point can be isolated.

Daya Visualization is a great method of building an understanding anout the data being used for forecasting. It is helpful for Analyst to review forecasting data using; - multi-dimensional plotting graphs (buble graph, 3d column or dot graphs). You can always find clusters that exist in isolation or on extreme ends and start there and move towards the crowd. - Related Data Trend lines may also help in identifying correlation candidates, or anomolies that are due to missing data or just one off event. Examples like dip in sales is normally seen with dip in traffic, or stock availability.

You can identify outliers in different ways: Classical methods: IQR, Zscore, Quantiles, but these are not the most relevant in a time context. Defining a confidence interval around a forecast or using anomaly detection algorithms may be more appropriate. Nevertheless, it's much simpler to procure stock, promotions and events to see whether these anomalies are anomalies or economic events with an identified cause!

Deletion: This is the simplest method, suitable when outliers are clearly due to errors in data collection or entry (e.g., typos, incorrect units). Winsorization: This method is useful when you want to retain the information contained in the outliers but reduce their extreme impact on your analysis. Transformation: This approach is helpful when your data is not normally distributed and outliers are skewing your analysis. Imputation: This method is suitable when outliers represent missing or unreliable data. Robust Forecasting Methods: This is an alternative approach that doesn't directly remove outliers but instead uses forecasting models that are less sensitive to them.

Filter Vs Delete: Not all data is junk data and at times it is important to us filtering technique to remove outliers and putting your new data to test of accuracy and true representation. Delete (remove column / row) is to be used when you want to remove junk data entries. Examples like dummy phone numbers on customer record. Data with partial values like Year of Birth, or zip code should be considered for Filtering technique.

Understand the Cause of Missing Values: Before imputing, investigate the reason behind missing values. If the missingness is not random, it could introduce bias into your forecasts. Choose the Right Imputation Method: Select a method that is appropriate for your data type, amount of missing data, and the underlying missing data mechanism. Sensitivity Analysis: Perform sensitivity analysis to assess how different imputation methods affect your forecast results. Document Your Decisions: Keep a record of which imputation methods were used and why, to ensure transparency and reproducibility.

Each error metric function characterizes your errors differently. While MAE is not very sensitive to outliers; RMSE will be penalizing outliers a lot more.

Split Your Data: Divide your historical data into two sets: a training set and a validation set. Train Your Model: Develop your forecasting model using the training set. Validate Your Model: Apply the trained model to the validation set (data the model hasn't seen before) and compare the predicted values with the actual values. Assess Accuracy: Use error metrics like MAPE, MAE, or RMSE to evaluate the accuracy of your forecasts on the validation set. Iterative Training and Validation: Train your model on the initial periods and validate it on the subsequent period. Then, add the validated period to the training set and validate on the next period, and so on.