Tuning a Statistical Forecast Part 3: Strategic Elements

Strategic Elements

The key pillars of a statistical tuning program are Data & Information, Systems, Methodology, Resources and Strategy. These elements need to be analysed in partnership so that your tuning goals can become aligned and made achievable.

• Data: (What) & Information (What is bad & What would be good).
• Systems: Where do you do the tuning?
• Methodology: The 9 steps of actually doing the tuning.
• Resources: Who can do the tuning.
• Strategy: What are your Business objectives?

A. Data and Information

Data drives your forecast creation and indicates your forecast accuracy. What information and accuracy do you have and what would you like to obtain? Here are 4 aspects to consider:

Accuracy Results. Naturally, you must first clarify how your statistical forecast is performing and presumably, if you're reading this then things aren't that good! So, where is it good and where is it bad?

In an ideal world you'd want to make every combination better but in reality, this may not be possible. Consider trading accuracy between combinations if it means that intersections of high volume, value & margin get a better baseline. Knowing what you can achieve will take some time but don't forget to take a snapshot of your forecast before you start tuning (and at regular intervals) so that you can measure your progress.

Accuracy Method. It is also important to validate your measurement approach. What exactly are you measuring (most recent forecast against most recent actuals or a 3-month moving average? or forecast against last year?). What calculation are you using? There are no right answers but there are more appropriate and/or less flawed ways to measure accuracy and these will depend on the data that you are using. Prepare to re-consider your accuracy measurement approach once the system components and data are defined for your tuning.

Accuracy Expectation. You must clarify the definition of “What is Good?” with regards to accuracy attainment. There are two aspects to this: first is the general target of accuracy to aim for (and the levels this will be measured at) and second is the achievement during tuning activities. Don’t tune forever in order to get to the magical "good", make it better & move on to the next method until you have explored all the methods and can use them effectively.

Data Availability & Suitability. The final aspect to consider is intrinsically tied with the Systems Components section below - what is your tuning data source & how long will it exist and be useful? The answer to this question will actually be the significant driver for most of the tuning activity that you undertake.

B. Systems

Take into account your available systems. You probably have Production, Training, Development & Support available. But which of these would be the best place to conduct tuning? The answer is none of them. A special Tuning environment is the best place, but given a dedicated environment, what data should it contain?

Many would assume the most recent actuals, but then how do you analyse the forecast for next week, next quarter or next year? Who decides which output is best (tuned) and how? If your data is volatile or has changed dramatically in the recent past, contemplate using a standard deviation or Budget as your baseline but beware internal bias.

A little cheat that I like to employ is to consider turning system time into the past so that you can create forecasts and compare them to actuals along a full or partial forecast horizon. Now that you are thinking about the significance of the environment, the data that it might contain and how you can use it let's evaluate the system components a little more:

• Production Environment. This will be the most up to date, the fastest and the easiest to use but the planners may not have access or 'rights' to perform some of the tuning methods. More significantly, tuning in Production is very risky! The best-case scenario is that the forecast gets better but the worst outcome would be that the system falls over due to the extra workload. If you run a monthly cycle from a single forecast run, tuning will be difficult to achieve without moving the official run date to allow for iterations and validation.
• Development / Training / Test / Support. These are standard environments that are likely to be available. Whether they are free for use in tuning is another matter. Some tuning tasks are relatively non-invasive (changing history or turning combinations off) but some others require configuration alterations and / or system re-booting. Perhaps not ideal if other critical development, training and support activity is taking place.
• A dedicated Engine Tuning Environment. This is really the best option as it will provide unfettered access and won't be limited by other activities. It will enable the planners to roll-back time and perform other configuration work that would be entirely impossible on Production or Support systems. The downside is that over time the data in the system will become old. Applying extensive changes into a Tuning environment to facilitate tuning (worksheets, tables, archived iterations & calculations for comparisons etc.) may have to be re-built in a refreshed environment.
• Engine Tuning Laptops with Virtual Machines. This approach is really useful for training since multiple users can access the system and run their own engines at the same time. Also useful for performing the more extreme types of what-if tuning analysis that might not be desirable on a full dataset or in a live system. Disadvantages of personal virtual machines is that they will require a restricted dataset. Cloud versions of this approach are probably the best approach since they may not require data restriction.

C. Methodology

The ways that you can change the statistical forecast are grouped into 9 steps that are in a rough order of complexity. The 9th step is management of the tuning methods and can become a complicated task in it's own right. The Easy methods are probably being done by planners now (though perhaps not in a holistic tuning way). The Moderate can probably be completed by planners with some training and the Complex conducted after more advanced training. Alternatively, employ 3rd Party companies to perform the tasks on your behalf.

1. History: Clean & adjust to create the best forecast. Difficulty Factor = Easy
2. Decision Process: Lifecycle / Segments / On or Off. Difficulty Factor = Easy
3. Hierarchy Levels: Edit to drive better usage. Difficulty Factor = Moderate
4. Engine & Models: Selection in engines or plans. Difficulty Factor = Moderate
5. Engine Settings: Tweak for optimal settings. Difficulty Factor = Complex
6. Causals: Add, remove, edit to obtain better results. Difficulty Factor = Moderate
7. Proport: Assess allocation & aggregation methods. Difficulty Factor = Moderate
8. Nodal: Refine above per individual combinations. Difficulty Factor = Complex
9. Tuning Methodology Management. Difficulty Factor = Easy to Complex!

Fuller details of these steps can be found in Statistical Tuning Article 2 but the relevance here is that a really tuned engine will need all of these active methods optimised. If you can't perform some tasks due to maturity skill (not trained enough) or system capability (restricted functions) then your strategy and expectations should be adjusted accordingly.

D. Resources

So, who should, can or will perform the tuning? The options are covered in fuller detail in Statistical Tuning Article 1 but let's briefly review the three options again:

1. Outsource: will enable all of the tuning methods to be employed (plus any special capabilities brought to bear by the 3rd Party.). This approach will be completed faster, cost more and have no guarantees. Nor will you retain any of the knowledge - just an improved forecast. There will be less bias or emotion though you will need to provide access to systems and data.

2. Use Demand Planners: Will cost less but take longer and likely not use all the methods available. This approach can overload already stressed planners but, if managed properly can provide excellent opportunities for learning and extend maturity growth. The Planners will know the data better than 3rd Parties - they will already know the areas that need the most urgent tuning.

3. Use Engine Tuners / Data Scientist: This approach enables even greater resource development. More training and perhaps a specialist department performing all kinds of data analysis. The Tuners will likely have less product or customer bias which will create a more accurate baseline. Ownership of the expertise will be retained in the company and the tuning can be applied regularly and potentially within the standard cycle.

E. Strategy

Having evaluated your current accuracy levels, the system to test in and the resources who will perform the tuning methods it is time to start setting some achievable targets and plan for maturity growth to enable future tuning.

It is unlikely that you will be able to utilise all the strategic elements to their maximum potential from the very start, in which case create a phased approach. This works for all the elements whether incomplete data, resource skill level or system availability. Start simple and progressively work to a higher level of competence and capability.

There are more factors to consider. You will need to start understanding a little more about Planning Maturity, Resource Availability and Company Objectives. More on those factors and how they impact your tuning journey in coming articles. Stay Tuned!