The language illusion, doubled

Is programming a computer more like language or more like maths?

Neither, it turns out. In recent research, neuroscientists at MIT conducted brain scans of programmers while they were trying to solve problems, and discovered that, rather than engaging the language centres of the brain, they engaged a system known as the multiple demand network, usually used for complex problem solving.

Programming languages, it seems, are not the same as ordinary languages. This is not new news. In the earliest days of programming, when Grace Hopper was inventing high level languages, she and her team sent versions of their code to their bosses in French and German. The bosses sat up and paid attention: was it possible that their computers had suddenly learnt to speak foreign languages? Of course, that was not the case: the vocabulary of the programming language was entirely arbitrary: as long as the compiler could recognise the symbols and convert them into machine code, then they could be whatever the team wanted them to be. The team chose English words because they made it easier to understand (and because they were English speakers), but it was simple to tell the compiler to look for different symbols.

The difference between programming languages and human languages, despite their superficial resemblance, can be an obstacle to people learning to code for the first time, or for people using tools which resemble programming languages, such as spreadsheet macros or low code platforms.

When we speak to humans, precision is useful but not mandatory. Our speech is often very imprecise: circular, allusory, full of hesitation and digression. Yet other people can figure out what we mean: they interpret, and empathise, and fill in our gaps. By comparison, a compiler or code interpreter is brutally unforgiving. We cannot indicate vaguely what we mean: we must spell it out in excruciating detail, and any mistake is punished with error messages or bugs. Computers are frustrating because they do what we say rather than what we mean.

But perhaps this is changing. We now have new tools: the Large Language Models (LLMs) that sit behind popular services such as ChatGPT. The attraction of these tools is that they seem to speak our language: we type (or speak) in plain English, and they write (or speak) in the same way. We can even get them to write code, or, through the agents that are increasingly populating the market, invoke code that already exists. Does this mean that we can stop getting frustrated with computers, and get them to do what we mean rather than what we say? Can we stop trying to satisfy the brutal precison of the compiler and the interpreter and simply tell the computer what we want?

LLMs are valuable new tools which allow us to do things with computers that we struggled to do before. But their apparent facility with language can create illusions as powerful as the illusion that a programming language is the same as a human language.

Some of these illusions are obvious and well documented. Humans have evolved to ascribe agency to things that talk to us, and sometimes users of LLMs have come to believe that the models are conscious. Marketing which describes models as ‘reasoning’ or ‘thinking’ does little to dispel this illusion. If you believe that you are immune to it, consider the difference between the way that you type terms into a search engine and prompts into an LLM. Have you ever ended your prompt with a question mark? Or said, ‘please’?

The consequences of this illusion are best left to psychologists and philosophers: in the meantime, there is another, more mundane illusion that we should be alert to. That illusion is generated by the same unforgiving precision that applies to coding languages. Despite our frequent frustration with computers, we have become used to them providing precise and accurate answers. When we provide them with the same input, we are used to getting the same output (and if we get different output, we know that the data has changed, or that there is a bug). If the record we are looking for is missing from the database, we expect to get an error message.

But this is not how LLMs behave. We can provide them with identical input and get different output. If the data is missing from the model (or from the datasets the system containing the model has access to), then we might get an error message - or we might get fake data which looks convincing. When we give LLMs instructions, they may or may not do what we say - a phenomenon familiar to anyone who has tried to get an LLM to produce a certain style of output, and gone through many rounds of prompts with limited success. Sometimes it feels as if we are pleading with an LLM to do what we want, in the same way that we would plead with an old-fashioned computer system that just keeps saying, ‘No’. We talk about prompt engineering as if it gives us as much control as software engineering, when it is really an exercise in iterative persuasion.

Programming languages are not like human languages: neuroscience tells us that they are something different, something closer to a puzzle. And the language in which we converse with LLMs is neither like a programming language, nor like human language. In both cases, it serves us well to understand the underlying mechanisms which give rise to these language like behaviours - and to learn how to control the tools which we have invented for ourselves.

(Views in this article are my own.)