Process > LLMs
What makes a generalist?
It is easy to think of generalists as jack of all trades and masters of none. Generalists are the kind of people who can do two wildly different jobs from one week to the next because their skills, experience, and interests provide a broad foundation of knowledge.
When viewed through this lens, LLMs are generalists. They have consumed much of the internet and can offer confident-sounding answers to almost any question.
The temptation to treat LLMs as generalists has fuelled much of the sensationalist claims that entire workforces can be replaced with AI. When much of the work in a technology-focused role is simply creating and moving information around, why use a human when an LLM can respond so much faster, with much more confidence, and never complain?
I’d argue though that measuring generalists by the breadth of their knowledge is the wrong metric. Instead, it is better to measure generalists by their ability to deliver outcomes in the absence of well-defined processes.
These two ways of defining a generalist overlap to such a large extent when applied to people that the difference is mostly irrelevant. Take the most common representation of a generalist: the full-stack developer. A full-stack developer has an understanding of frontend and backend development, database management, cloud technologies, deployment strategies, and monitoring because they have done all of these things in the past and can replicate them in the future. It is expected that having and applying knowledge are two sides of the same coin. Seasoned full-stack developers can deliver meaningful outcomes with very little established process as they establish enough of a foundation on which to build the desired result.
However, these two definitions of a generalist can not be applied to LLMs. LLMs have a breadth of knowledge that no human could ever hope to attain. However, they can not apply that knowledge without rigorous processes.
If you ask an LLM to generate a web app for a shopping cart, it will do so and will produce something at least as functional as an average developer would produce. What an LLM will not do is infer your intention and also:
领英推荐
In other words, an LLM will do what you ask it to do, but nothing more. Indeed, LLMs won’t be able to implement anything other than a text-based response without a serious amount of work to integrate it with all your platforms.
If you asked a human full-stack developer to produce the same web app, they would intrinsically understand the full scope of what was being requested and have the ability to deliver a complete solution.
When measured by their ability to deliver meaningful outcomes in the absence of robust and repeatable processes, LLMs are not generalists. They have amazing potential, but only when existing processes provide them with all the input they need, and can consume and implement their outputs.
The most successful implementations of LLMs will come from teams that have already done the hard work of building robust, repeatable, and trusted processes.
I expect though that many companies will presume that LLMs are generalists because they produce confident, and ever more accurate, answers to almost any question. This, in turn, leads to the idea that they now have an infinite supply of tireless robot employees ready and eager to turn high-level ideas into solutions as fast as a GPU will allow it.
It won’t work out well.
Completely agree Matt!