My two cents on LLMs and Optimization

I’d like to weigh in briefly on a Harvard Business Review (HBR) article by former colleague, Prof. David Simchi-Levi, and some criticism it received on social media from Joannes Vermorel (and from others who piled on in the comments section).??

Before I get into specifics, I’d like to state that the insulting tone of Joannes's piece is uncalled for. There is no need for phrases like “drivel,” “lazy,” and “bullsh*t.” They don’t inform anyone and only lower the quality of our discussion.

I’m going to focus this article specifically on the use of large language models (LLMs) for optimization modeling. This is an area that David and I collaborated on for many years, and thus I can guarantee Joannes that we do indeed have experience.

I agree that the HBR article is optimistic. To the extent that it implies non-technical people can use LLMs to modify Mixed Integer Programs, it’s just wrong. That said, I’m not entirely sure that’s a fair implication. The HBR article acknowledges that LLMs make mistakes. Specifically regarding math modeling, the article states that automated verification of correctness is an open question. I took this to mean the proposed use of LLMs, at least in this area, would be under the supervision of a technical expert (i.e., a math programmer) who would bear ultimate responsibility for correctness.

It certainly seems reasonable to me that an LLM would be able to study a codebase dedicated to mathematical optimization and create the proposed changes for adding a new constraint or objective function component. For example, a few years ago I created a short Python course for building strategic network design models. Recently, I’ve fed the exercises from the course into ChatGPT (both o1 Pro and 4o) and the results were impressive. 4o created a few bugs and o1 Pro was nearly perfect. So the idea that an optimization engineer would use an LLM to more quickly address such a task strikes me as realistic, not science fiction.

I’ll add a caveat here: so long as the optimization engineer is using a very popular programming language. An LLM is not going to be equally skilled with every language. The more online examples and discussions exist on the internet, the more training data is available to the LLM. This inevitably results in the LLM’s preference for languages that rank high on the TIOBE index. For my specific exercises, the LLM was programming in Python, the most popular language in the world.

This brings me back to the part of Joannes's article that strikes me as less than forthright. Joannes touts the benefit of his domain-specific language which is allegedly 10x more concise than Python. Considering that Python often has the brevity of pseudo-code, I’m not sure such a claim, even if true, is a net benefit to the maintainability of a codebase. In the words of Einstein, something should be “as simple as possible, but not simpler.” My assessment is that Python itself has about as much brevity as anyone wants in a programming language.

Of course, maintaining fewer lines of code is a net benefit, and Python programmers typically achieve this goal through libraries. The Python open-source ecosystem is famously vast, and for domain-specific tasks like supply chain it can be enhanced by proprietary libraries maintained in private repos. I expect Lokad boasts a rich set of such libraries, or has functionality that could be supported in a library built into their language proper. So I’ll grant Joannes his claim that for certain tasks he can achieve a 10x reduction in codebase relative to a Python solution that is supplemented only by the PyPI libraries. I don’t think it's fair to say such a reduction is guaranteed against any Python solution. The history of Python, after all, is rife with successful reverse engineering of libraries that originated in other languages.?

This is all a bit of a digression from the larger point at hand, which is the role of LLMs. I think it's a fair assumption David is proposing that LLMs be used to address programming tasks for codebases written in languages very high on the TIOBE index. As I mentioned earlier, I have no expectation that ChatGPT is as fluent in Lokad’s language (called Envision) as it is with Python. The size of the training data here simply cannot compare.?

This brings me to my final point. Regardless of whether you are an LLM optimist or a skeptic, I think it’s safe to say that LLMs are bad news for niche proprietary programming languages. As companies move from spreadsheet logic to codebases, tools like ChatGPT will contribute to the same network effect that websites like Stack Overflow did ten years ago. If you are using a popular language, you get better support, which increases usage of that language, and thus improves the support for those that follow.?

I should conclude by putting my own cards on the table. I, too, work for a software company in the supply chain space that helps clients move toward programmatic solutions. Our libraries and our apps are nearly all in Python (with some Rust scattered about for performance). Our clients often run those solutions on our web-based platform (Foresta). As needed,?these solutions are run in their own playground, without our supervision. Our clients are comforted by the knowledge their business logic isn’t trapped in a programming language owned by anyone, nor is it caged by a platform that guards against portability.?

And now, with the rise of LLMs, they are further comforted that their business logic can be explained to them by an affordable automated tool. Five years ago, such an idea was science fiction. Today, it is real.