LLMs & World Models: Bringing Focus Back to the Real World

The next phase of the internet isn't about getting even more lost in digital worlds or escaping reality through our devices. Instead, it's about making the physical world itself understandable to machines. This shift brings our attention back to the real world around us. By enabling computers to interpret and interact with our surroundings, we're laying the groundwork for technology that enhances our everyday experiences rather than replacing them.

To understand this transition, it's essential to examine the evolution of the consumer internet. In its first phase, the focus was on making webpages machine-readable, enabling algorithms to generate and organize content for human consumption. Google emerged as the clear winner in this era, building a powerful knowledge graph and a formidable advertising business.

The next phase saw a shift towards making people machine-readable, enabling algorithms to generate and organize networks of individuals. Facebook, taking cues from Google's success, built another advertising giant by leveraging a social graph powered by mobile, social, and cloud computing. Both these approaches made life into a movie, a spectacle we all viewed, commented on, liked and disliked. This is our version of the Roman Colosseum.?

But, we're entering a new phase where reality itself will be made machine-readable, permitting algorithms to understand and interact with the spaces in which we live and move. This transformation is driven by advancements in computer vision, spatial intelligence, and smart sensors, all powered by machine learning and neural networks that can comprehend movement in physical space.?

While some may attempt to capitalize on this shift through advertising, as seen in previous phases, the true potential lies in creating technology that enhances our everyday experiences and interactions with the world around us.

Central to this transformation is the map, evolving beyond a simple navigation tool into a dynamic platform powered by spatial intelligence, or spatial AI. Think about it this way. If you and I are together and we both look at our Facebook feed, we’ll see completely different content. Should we engage with that content by liking, commenting, sharing, etc. we can refresh the page, and we’ll immediately see a new feed! But, if we both open Google Maps, we’ll see the exact same map. This despite our worlds, interests, movements being totally different. What if the map were alive like our feeds? Using machine learning and neural networks to have the map respond to you and your actions to deliver a personalized, smart map. You see your map.

This technology allows smart agents to work within our physical environment, handling tasks for us and enabling effortless interactions with the world we live in. However, it's crucial to recognize that our relationship with the world is always mediated by various factors such as concepts, language, culture, experience, and technology. Often, we mistake the map for the territory, confusing the screen and what's displayed on it for reality itself.

While it's not about introducing an unmediated experience of reality, we have the opportunity to design a different kind of product, one that acknowledges the extent of mediation and the reasons behind the kind of mediation we see from tech giants. Instead of focusing on an "unreality industry" or replacing the world with the screen, we can create a product that works to attune us to our world, keeping us in sync with it. This is the potential for spatial computing—a design revolution that prioritizes harmony between technology and the human experience.

However, as we imagine this future, it's crucial to address the concerns and objections that arise with the advancement of AI and machine learning. The rapid development of these technologies has led some to worry about the potential irrelevance of the human species in a world dominated by runaway intelligence. This fear has even prompted calls for a deceleration of innovation in AI and ML research.

Throughout history, humanity has faced paradigm shifts that have challenged our place in the world. Copernicus displaced Earth from the center of the universe, Darwin's theory of evolution by natural selection humbled our position among living beings, and Freud's concept of the unconscious revealed that we are not entirely in control of our own minds. Now, AI presents another paradigm shift, where intelligence is no longer confined to an organic substrate but can migrate into silicon or machines and evolve at rates beyond the human brain's capabilities.

In light of this, we must ask ourselves: what is the role of human beings in a future of intelligent agency? What kind of world are we envisioning? Is it a future like the dystopian movie Wall-E, where humans become passive and dependent on technology? Or can we design a world in which the symbiotic relationship between human and artificial intelligences leads to human flourishing rather than increased suffering?

As we embrace this new model of spatial AI and map interfaces, we must actively shape the future to ensure that technology supports and empowers humans rather than diminishing our role. We need to move past the "one-size-fits-all" approach that often reduced diverse experiences into cookie-cutter interactions and instead recognize that place, time, and culture matter. Technology should adapt to these nuances to provide more relevant support while respecting the uniqueness of human experiences.

The next phase of the internet should be about action and movement, not just consuming information. It's about using technology to strengthen our connection with the physical world rather than distracting us from it. By making reality machine-readable, we open up possibilities for richer experiences, greater convenience, and a more natural relationship between humans and technology.

However, we must approach this future with a clear-eyed understanding that no technological advancement leads to a utopia. Every development comes with its own set of challenges and potential drawbacks. As we work towards a future where intelligent agents operate in the world on our behalf, we must prioritize human agency, creativity, and well-being.

The role of technology should be to augment and support human capabilities, not to replace them. We must design systems that enhance our ability to engage with the world around us, to learn, to create, and to connect with others. By keeping human flourishing at the center of our technological progress, we can strive for a future in which the symbiotic relationship between human and artificial intelligences leads to greater understanding, empathy, and prosperity.