Why Generative AI Lacks True Understanding of the World

Large language models (LLMs) have demonstrated remarkable capabilities in recent years. They can compose poetry, draft code, and even give accurate directions — skills that suggest they might be building an understanding of the world. However, recent research challenges this notion, revealing that LLMs, despite their impressive outputs, may lack a coherent, structured understanding of real-world concepts.

Limitations in Learning the "World Map"

A recent study focused on a widely used generative AI model known as a transformer, the core architecture behind models like GPT-4. Transformers are trained by predicting the next word in a sequence, allowing them to produce realistic text across various topics. This method, however, does not mean the model grasps the underlying structure or "rules" of its subject matter.

The research team tested this by giving the model navigation tasks in New York City, where it accurately provided turn-by-turn directions. Yet, when they introduced street closures and detours, the model's navigation accuracy dropped dramatically. The internal "maps" generated by the model also included several fictional streets, revealing the absence of a coherent, accurate spatial understanding.

Exploring Generative AI's World Model with New Metrics

The study introduces two novel metrics to evaluate these generative AI systems' "world model" quality. Traditional metrics focus on prediction accuracy. However, the researchers aimed to assess whether the models formed structured understandings of real-world scenarios. Their new metrics are:

1. Sequence Distinction: Measures whether the model recognizes different states in a sequence. For instance, can it distinguish between two different positions in a game of Othello?
2. Sequence Compression: Tests if the model understands when two identical states lead to the same set of possible next steps.

Using these metrics, the researchers examined two types of transformers: one trained on random sequences and another on strategic or patterned sequences. They found that transformers trained with random choices developed better world models, likely due to exposure to a broader range of potential outcomes.

Incoherent World Models: Implications for Real-World Applications

In testing, the researchers discovered that even though models could generate valid Othello moves or accurate directions under normal conditions, they failed when the scenario deviated from the usual setup. For instance, a slight change in the New York City map led to a steep drop in navigation accuracy — from 100% to 67% — after only 1% of streets were closed.

When the researchers visualized the internal city maps generated by the models, they found many imaginary streets and erroneous structures, such as streets overlapping or in impossible configurations. This result highlights the model's need for proper spatial understanding despite its apparent proficiency in giving directions.

What Does This Mean for Generative AI in Science and Technology?

This discovery has significant implications for generative AI's role beyond language tasks. Suppose transformers can't build reliable world models. In that case, they may need to catch up when applied to scientific and technical fields where real-world understanding is crucial. According to Ashesh Rambachan, an assistant professor at MIT and senior author of the study, "If we want to use these techniques to make discoveries, we must ensure that these models understand world rules in a structured, accurate way."

Future Directions: Building Coherent World Models

The researchers plan to expand their work to include more complex problems, especially those where rules are partially known. By refining these evaluation metrics, they aim to design models that can tackle real-world scientific challenges more effectively. Their goal is to improve AI systems' "world model" to ensure they can perform consistently, even under unexpected changes.

Generative AI's progress is impressive, but these findings reveal an essential limitation: predictions and performance do not necessarily imply understanding. As we advance in AI technology, examining and addressing these gaps in coherence is crucial if we aim to apply AI in science, navigation, and other real-world fields.

Funding and Acknowledgments

Grants from the Harvard Data Science Initiative, the National Science Foundation, the Vannevar Bush Faculty Fellowship, the Simons Collaboration, and the MacArthur Foundation supported this study.

Key Takeaways

• Generative AI, especially transformers, can perform complex tasks like navigating or generating game moves without understanding underlying rules.
• Introducing new metrics — sequence distinction and sequence compression — provides insights into whether a model has a coherent world model.
• Models that appear accurate in stable environments may falter under changing conditions, raising concerns about deploying generative AI in unpredictable, real-world scenarios.

The study underscores a critical need for AI development: building models that predict and genuinely understand. For AI to reliably contribute to fields beyond language, it must develop a structured comprehension of the world's complexities.