The Great LLM Debate: Does ChatGPT truly understand?

A fascinating and heated debate continues to rage about the true utility of Large Language Models (LLMs) like #ChatGPT. Some argue these models don’t really understand anything they are writing. A vocal critic, Gary Marcus , makes the case that LLMs are just really good at paraphrasing the text they have been trained on. They give the impression of understanding and of some level of intelligence, but it’s all just a fa?ade.

The alternative view is that the skill in manipulating language, which an LLM such as ChatGPT clearly displays, is evidence of both genuine understanding and of (a form of) intelligence. It's easy to find examples demonstrating LLMs performing tasks including:

• summarising and paraphrasing text;
• writing short blocks of code, and translating code from one language to another;
• writing poetry and song lyrics; and
• brainstorming ideas (for recipes, business plans, article headlines, etc).

In many cases, the text produced is not just coherent, but contains complex sequences of logic. It seems somewhat far fetched to suggest that these are created merely by statistical pattern matching without any understanding of the words and phrases used.

So where does the truth lie? What level of understanding do LLMs have? Are they simply stochastic parrots as described by another leading critic, Professor Emily M. Bender , paraphrasing based on nothing but a highly sophisticated knowledge of word and phrase co-occurrence statistics?

Perhaps this doesn't matter? One counter argument is that even if this were true, if an AI produces text which is reliable most of the time, we can probably find uses for it. Let’s not worry unduly about whether the AI actually knows what it is saying if it gets the answer right most of the time.

There is undoubtedly some truth in this: lots of people have noted that ChatGPT and/or Github co-pilot have significantly improved their coding productivity - saving time looking things up on Google / Stackoverflow.

The difficulty is that unless you either already know the expected answer, or at least know enough to verify the results, how can you trust what the LLM says? By way of example, see the following exchange I just had with ChatGPT:

It’s easy to read the output and assume that ChatGPT ‘knows’ quite a lot about US presidents, but just made a mistake. But this is very much called into doubt by the second answer. This isn’t merely a factual error: it shows a fundamental lack of understanding of the text that is being written.

For the time being, the conclusion appears to be that whatever level of understanding LLMs have, it is severely limited. As such, the practical uses for these tools will (or perhaps should) also be constrained. Certainly, LLMs don't appear ready for business critical tasks.