Thinking differently: Avoiding Optimisation 1/2
A lot of business people and analysts are obsessed with optimisation; the best possible business strategy; a single point which best satisfies some set of conditions;? for example the product mix which gives you the greatest profit. Optimisation offers a manager the scientifically proven best thing to do. The unarguable answer. To have a machine which automatically gives you the Optimum result? That’s got to be a good thing, right?
I’m not so sure, for two reasons.
Firstly, it gives you the right answer only in a very narrow sense which may not suit your particular problem. And secondly, the certainty with which it infuses the decision process ends up stifling proper engagement with your business problem. I am going to focus on the first issue in this blog, and on the other issue in a future blog.
So here’s a question: Is the “Optimum” you are being given, the result you really want?
And here’s another: There’s more than one optimal result? Surely that’s a contradiction in terms?
Nope.
When people talk of the optimum, they normally mean the global maximum of your function. In other words the set of conditions for which the function’s value cannot be increased. Normally you set the function in question to be your profit, so the optimised function is the biggest possible profit. Seems sensible enough??
This is the maximum result, but it is only the best result for your use case if that’s what you want, and while maximum profit sounds like something you always want, ceteris paribus, keep an open mind and read on.
1. Knowing what the “optimum'' is doesn't necessarily help you get there.
If in real life you can’t set your conditions arbitrarily (as you do when you enter them into your computer calculation) then the calculated optimum may be a big step away from what you are currently doing. You might ask yourself whether it is too big a risk to completely change your business model in the hope that your calculation is right. Well punk, do you feel lucky??
For example, if your optimisation model told you that, due to the elasticity of demand, you could halve production and double prices, would you do it?
Instead you could try: Marginal improvement: what is the one next best action you can take given what you are already doing. Not only is this safer, but over a sequence of marginal gains you may find that your business model changes in a way that wasn’t anticipated in the optimisation.
2. Getting to the optimum might be prohibitively disruptive or slow.
This is the 80/20 thing. It's all very well knowing the optimum, but you might be able to get most of what you need without going all the way, and a conventional optimisation model isn’t going to tell you what that will be.
For example, imagine that profit on a non-core product could be doubled with a lot of focus, or could grow 50% without much effort, but in either case it would only represent 1% of total EBIT.
You could try: Local optimum:? what is the smallest possible change you can make to meet your objectives given what you are doing? In this case you might say what is the easiest, least disruptive way of getting where I need to go?
领英推荐
3. Getting the optimum result might be unlikely or risky.
Single point outcomes are deceptive, because it is vanishingly unlikely you will ever achieve exactly that value: it will most likely be a bit above or a bit below. Optimisers often ignore the trade off between risk and return.
You could try: Confidence interval:? what is the best possible result that you can be sure of attaining to a set level of confidence? An added advantage of this approach is that you can put in place a range of actions to address the range of potential outcomes. So in a supply chain contract, you could put in place firm orders for the number of parts you will need with a 50% probability, and a flexible contract for the next 30%, and an incentivised contract for the final 20%. The structuring of incentives and behaviour will be more important to the outcome than knowing what the exact theoretical optimised result is.
4. Your competitors might have something to say about what you are doing
You have no idea what your competitors will do, but it's not rocket science to assume that they aren’t going to sit back and let you optimise. They will react, and in a zero sum game (the simplest setup), their gain is your loss.
What if your optimised action forces your competitor to double manufacturing capacity, and that if they do that they can then half their costs? You might have been better not to provoke them. Sometimes your best action might be the one in which no matter what your competitor does, you still earn a good profit.
You could try: Minimax:? what is the action that you can take so as to give you the best possible outcome if everything out of your control (including the actions of your competitors) goes against you?
5. And you might want to change your response when you see what they are doing
This is a more complex version of the zero-sum setup above. If you can guess what your competitor will do in response to your actions, there may be a result that is better for both of you than the defensive Minimax result above (Of course it may be hard to communicate this without illegal market coordination). The point is that you can probably do better than the purely defensive Minimax result if you consider that the competition will act to avoid confronting you in exactly the same way as you will want to avoid provoking them.
You could try: Nash equilibrium:? if your competitors were able to observe your actions and could optimise their response, and you could do likewise with respect to their actions, what would you both do?
6. You might fail before you get to the optimum
In some ways this is a version of the first point. If the path you take to get from where you are to an optimum of untold riches has a good chance of making you bankrupt before you get there, you may choose not to go that way. Never ignore path dependency.
For example, many businesses have tried to increase returns by borrowing (leveraging the balance sheet) and have ended up in restructuring. Perhaps they should have spent less time optimising their Excel models and more time thinking about one of the alternative optima above.?
You could try: Path to equilibrium: If it takes time and a number of steps to get from your current position to the optimum result it is worth building an optimiser which takes time into account alongside the state (eg the balance sheet). You may find that the path takes you to outcomes which are different to the overall optimum.
I hope I have shown you that you need to think more about optimisation models, and the best outcome is often not the global maximum.
Next the other problem with optimisation: It makes you vulnerable in a fast changing world.
Head of Data Science @Riachuelo
3 年Technically, I think that some of the presented challenges might be tackled by multi-stage scenario-based formulations, for addressing the "how to get to the optimum" question, or robust formulations, for incorporating uncertainty in the estimated parameters. Yet, optimisation is just another tool and decision-making is a lot more fluid than that. For instance, I would never leave my house without checking the Waze suggestions, but some roads are just too beautiful to miss and I don′t recall Waze taking travel pleasure as a possible objective in their algorithm, right?
He/Him. Black lives matter. Making the world a little better, one byte at a time.
3 年One of the most important things I learned from painting, not so much from data science, was how to tell when my work was done. You can always add more, you can always keep painting, you can always do one more brush stroke... but the ability to look at something and acknowledge that it is 95% of what you aimed for, celebrate that, and call it done, is something that has really helped me in my professional workspace.
I really enjoyed the reflection. People in general are amazed with the benefits that Data Science and AI can bring and then tend to be overly picky. It is not possible to model the whole business in an algorithm, we can't use an algorithm to define our strategy. Optimization like other models brings huge benefits for specific, well defined and very well controlled problems, optimization of human and machine resources, activity distribution, production planning, stock management, supply chain, routes... A model that combines this, that choose the most profitable products, positioning and pricing, which campaigns? and when? it's still science fiction...
Robotics Software Engineer @ Amazon | Expertise in Robotics and Automation
3 年Interesting article, thanks for sharing!?However, I think the main problem with global optimization is that it is hard! As a consequence, we need to do multiple steps of abstractions and simplifications as a prerequisite. In my opinion, most of the issues mentioned in the article arise when our final abstraction is missing some critical detail of the original real-world problem. To the extent that a solution to the abstract problem no longer applies to the original. Some of these critical details might be: 1) Dynamics The world is dynamic (changes with time). And most likely that an optimization problem arising in practice involves dynamic variables. If we ignore the dynamic nature of the problem, we end up with a steady-state solution. For example, if we are trying to maximize the profit, the steady-state solution simply tells us that when our system is at a state of maximum profit, how to set our controlled variables such that it remains there. Most likely, this has nothing to do with what we want to achieve! No wonder that when we apply the prescribed solution, we never reach the state of maximum profit. Probably, we want to find out that given our current state, how to set our controlled variables, not only in the present but in all points of time such that we achieve the maximum accumulation of profit. Solving such dynamic optimization problems is more involved and requires advanced methods such as optimal control theory. 2) Constraints Omitting constraints will result in easier optimization problems. However, the unconstraint solutions might not be realizable. For example, suppose we want to find out how to move from point A to B in the shortest time. If we do not penalize energy expenditure nor include it in the constraints, we will get a solution that costs an infinite amount of energy and is therefore unrealistic. 3) Exogeneuous variables Are there any external variables that are not under our control and strongly affect our objective? If yes, we need to include a prediction of these variables into our problem formulation or else our optimal solution might be invalid. The amount of supply created by a competitor (as pointed in point (4) in the article) is a typical example of an exogenous variable. Finally, a small change to an optimization problem (e. g. adding one additional constraint) might significantly change the global optima. Real art is finding a problem abstraction that preserves the essence of the original problem while being tractable.
Analytics | Data Science | AI | Decision-Making | PhD Researcher
3 年Wonderfully written Harry. Lot’s of real life here, not only lab life!