To Prompt or not to Prompt?

Bloatware is something I really abhor, and it has been a problem for a long time as we pile one technology layer on top of another. Expedience has always been the justification, and it is reaching its current (but certainly not lasting) pinnacle with the latest GAI offerings.

A simple, practical example from our development work: we needed a classifier that could determine whether a webpage is biographical or not. We needed a simple Yes/No answer. This could be tackled with a lean, dedicated, language-encoded model, or we could just feed the question to a large language model (LLM). This doesn’t require anything as sophisticated as the latest ChatGPT model. A self-hosted LLM like LLaMA 2 is perfectly capable of handling such a simple job. In fact, it’s already enormous overkill. Since it is designed to be verbose, our junior developer found that, although the LLM was specifically prompted to provide only a Yes or No answer, it quickly "wandered off the reservation." I suggested to him that he simply include the prompt in every request, and this seems to have solved the problem. However, I have to admit that I feel somewhat bad about it, because it’s an incredibly wasteful way to approach the task—alas, an expedient one.

It reminds me of The Turing Option by Harry Harrison. In this story, a researcher develops an advanced AI, but after he is almost murdered, the AI is stolen and altered, eventually ending up in a lawnmower-like machine used for mundane tasks like identifying and killing weeds and pests. (By the way, I had to use ChatGPT and Microsoft Copilot to retrieve the name of this book, which I read decades ago. The former first hallucinated a non-existent book called Mowerman, and the latter confused it with Stephen King's The Lawnmower Man, but got it right after another prompt.)

We now run a model that contains more knowledge than any single human mind, just to give us a simple Yes or No answer on a rather mindless task. It may not be as wasteful as tying up a GPU with crypto mining, but it certainly makes me appreciate that these feed-forward neural networks are probabilistic automatons without any sentience. Otherwise, who could ever blame them for a robot rebellion?