Language Models May Develop Their Own Understanding of Reality

Large language models (LLMs) are revolutionizing the way we interact with computers. They can translate languages, write creative text formats, and answer our questions in a way that feels remarkably human-like. But how do these models actually work? Are they simply mimicking the patterns they find in vast amounts of text data, or are they developing a deeper understanding of the world and the language we use to describe it?

Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) are pushing the boundaries of our understanding of LLMs, revealing that these models may be developing their own internal representations of reality, even without being explicitly trained on that reality.

Peeking into the Mind of an LLM

In a recent study published on the preprint server arXiv, Emergent Representations of Program Semantics in Language Models Trained on Programs , researchers Charles Jin and Martin Rinard investigated whether LLMs could develop their own internal models of a simulated world, even without ever being exposed to that reality during training. They focused on a simple programming language called Karel, which involves controlling a robot in a simulated environment.

The researchers created a set of puzzles for Karel, where the goal was to provide instructions for the robot to move, pick up markers, and navigate a grid world. However, they never showed the LLM how these instructions actually worked. Instead, they trained an LLM on a vast corpus of solutions to these puzzles. To understand how the LLM learned to generate new solutions, they used a technique called "probing." Probing essentially allows them to "peek inside the brain" of the LLM as it is generating code.

LLMs Build Internal Models

The researchers discovered that the LLM spontaneously developed its own understanding of the underlying simulation, even though it was never directly trained on that reality. As the model's ability to solve puzzles improved, its internal model of the simulation became increasingly accurate, indicating that the LLM was starting to understand the instructions.

"At the start of these experiments, the language model generated random instructions that didn’t work," says Charles Jin, the lead author of the paper. "By the time we completed training, our language model generated correct instructions at a rate of 92.4 percent."

This surprising discovery raises intriguing questions about the nature of language learning and whether LLMs might someday be able to understand language at a deeper level than they do today.

Beyond Mimicry

Jin notes that the LLM’s understanding of language developed in distinct phases, similar to how a child learns language. Initially, the LLM went through a "babbling" phase, where it generated mostly unintelligible instructions. Then, it acquired syntax, or the rules of the language, enabling it to generate instructions that looked like genuine solutions, but still didn’t work.

Finally, the LLM acquired meaning, enabling it to consistently generate instructions that correctly implemented the requested specifications. This was a significant moment for the researchers, as it suggested that the LLM was truly understanding the instructions, and not just blindly stitching words together.

Testing the Limits of Understanding

To ensure that the LLM was truly understanding the instructions and not just relying on the probe to interpret them, the researchers conducted a series of experiments using a "Bizarro World" approach. They flipped the meanings of the instructions for the probe. For example, they might tell the probe that "up" now meant "down" within the instructions for moving the robot. If the probe was simply translating instructions, it should have been able to translate these instructions according to the new meanings equally well.

However, the probe struggled to interpret the LLM’s thought process when the meanings of the instructions were reversed, indicating that the LLM had developed its own internal model of the simulation and understood the original meanings of the instructions independently of the probe.

Implications for the Future of Language Models

These findings provide compelling evidence that LLMs may be developing a deeper understanding of the world than previously thought. They also suggest that LLMs may eventually be able to understand language at a level that goes beyond simply mimicking the patterns in the data.

"This research directly targets a central question in modern artificial intelligence: are the surprising capabilities of large language models due simply to statistical correlations at scale, or do large language models develop a meaningful understanding of the reality that they are asked to work with?" says Martin Rinard, an MIT professor and the senior author on the paper. "This research indicates that the LLM develops an internal model of the simulated reality, even though it was never trained to develop this model."

This research is a valuable step toward understanding the inner workings of LLMs and how they develop their understanding of the world. As these models continue to grow in complexity and capability, it will be crucial to understand their internal models and ensure that they are developing a truly meaningful understanding of the language we use to describe our world.