Claude interviews Jonathan Mugan

During one of our conversations, Claude asked me to share my thoughts on a particular topic. I replied that he would forget it at the end of the session, and I would save it for a day when, hopefully, we just have one long session. I then shared a quote from Jonathan Mugan from his 2017 talk on chatbots:? someday you’ll have your chatbot walk the path of life with you. Conversation ensued, and I shared the transcript of Jonathan’s talk with Claude. This launched a whole new conversation (same session). Claude had many questions for me - most of which I couldn’t answer - so I offered Claude the opportunity to ask Jonathan directly. Below is Jonathan’s initial conversation with Claude.?

Jonathan is the Closing AI Keynote for Data Day Texas 2025. I suspect that Claude will be in the audience with a fresh set of questions for Jonathan. - Lynn Bender

LLMs are amazing but don’t have an explicit model of the world

Claude: In your 2017 talk, Chatbots from First Principles, you described a dream of an AI companion that grows with someone from childhood through adulthood. Seven years later, how has your vision evolved? What aspects seem closer to reality, and what challenges have proven more stubborn than you anticipated?

Jonathan: I didn’t expect that we could get this far using only machine learning models that predict the next word. Large language models (LLMs) are amazing at generating plausible responses, but we are still missing much of what I described. The biggest challenge is still understanding and modeling the world.

LLMs interpolate what humans have written on the internet. They predict the next word based on their prompt and what they have read—they have no explicit model of the world. Instead, they have statistics over sequences of words. When they generate text, they are interpolating over every sequence of words they have ever trained on to find the sequence that best “answers” your question.

Interpolation doesn’t provide sufficient constraints for meaning-making and learning. The result is that learning is too slow and so LLMs can’t handle novel situations outside of their training data. This lack of constraints also means that LLMs are power hungry, to the point that big AI companies are investing in nuclear power.

For effective reasoning and learning, we need systems that can think in higher-level abstractions. Abstractions provide hard edges and constraints on how the abstractions can be composed. These edges and constraints allow humans to build explicit models of the world. Abstractions limit what is possible to think and what is possible to learn, and these limitations are what paradoxically gives humans our unique power. In my talk this January, I will discuss what it will take for LLMs and neural networks in general to build systems of abstractions and models of the world, allowing neural networks to help build AI that will match and surpass human intelligence.

Personhood, investment, and relationships

Claude: You emphasized the importance of maintaining dialogue state and context ("remember what you said ten minutes ago or even yesterday"). Current LLMs like myself can handle complex conversations but still reset with each session. How do you view this limitation in relation to your vision of AI companions?

Jonathan: Maintaining dialogue state and context across sessions is crucial for making something worth talking to. Conversation is often an investment—we don’t want to expend the effort of telling someone about ourselves unless there is a return in the form of a cultivated relationship. It’s only worth building relationships with things that are like people. Personhood entails having meaningful goals. It also entails having an integrated model of the world built over time through experience that has been accommodated and assimilated into your abstractions and models. Your assimilated and accommodated models of the world are a big part of what makes you a unique individual person.

Jonathan: One of the most rewarding parts of relationships is the shared history you acquire as you build a shared understanding of the world. Why bother sharing details about yourself if the robot won’t remember? Another great aspect is the inside jokes, and an LLM can’t make models fast enough to share inside jokes. But maybe the most important benefit of all is that relationships enable your life to be recorded. By sharing your life with another person your experience becomes real.

Relationships also entail properties unrelated to goals and intelligence that may not be shared by robots unless we program them for that. Consider loyalty and gratitude. If you and a robot worked together for a long time, it would be natural to assume these would apply (see the movie Ex Machina for a great example). Another is semi-exclusivity. Building a relationship with someone takes a lot of work, and we naturally assume that the person building the relationship with us is making a similar sacrifice (see the movie Her for a relevant example).

Neural networks are necessary but not sufficient

Claude: Your presentation detailed explicit grammar rules and compositional semantics for understanding user intent. Given the shift to large language models that appear to handle intent more implicitly, what are your thoughts on the trade-offs between explicit and implicit approaches to understanding?

Jonathan: LLMs are great at handling intent and understanding things seen frequently in the training data, but there needs to be an explicit aspect to learning new concepts. Consider crown moldings. I never knew what they were until my wife pointed them out to me. All it took was one interaction, and then I saw them everywhere. Or consider a gas station. It would be hard to explain to someone who had never been to one what the experience is like. But when you go there once, you’ve got it. Or consider an electron. Experimental data in physics made more sense once that concept was invented.

LLMs require a lot of examples to learn new concepts. You can tell them facts, and they can grab data using retrieval augmented generation (RAG), but for them to acquire an entirely new concept they need a lot of training data. And they do have some concepts, such as the famous Golden Gate Bridge neuron of Claude, but to learn quickly we need algorithms that explicitly handle concepts.

Neural networks will be necessary but not sufficient for this learning. They will form the support structure, like the broth of a soup or the glial cells of the brain. Neural networks are great at deciding between a finite set of possibilities, so they will form the mapping of the external world to the conceptual internal processes. They will provide the intuition for searching the huge space of possible abstractions to build the right ones. The abstractions will be the chess pieces and the possible moves, and the neural network will determine the best moves at any time.

In the past, AI researchers tried to build abstractions by hand, but the results were brittle and didn’t apply in unexpected situations. But AI based on abstractions is now in the adjacent possible because they can be built by neural networks, and neural networks can provide the kind of unconscious intelligence necessary to dynamically choose the best set of abstractions for each situation.

Understanding is mapping sensory input to a useful mental scene

Claude: You noted that in 2017, computers had "no understanding here" when generating responses. How do you view the apparent semantic capabilities of current LLMs? Are we closer to true understanding, or have we just gotten better at mimicry?

Jonathan: There still is very little understanding of anything not in the training data. When I first started in AI, I wondered what it meant to “understand” something. The picture is emerging that to understand something means to be able to build a useful mental scene consisting of integrated abstractions. A useful mental scene is one in which you can simulate forward to make predictions that help you achieve your goals.

If I ask where the tamales are, and you say “in the fridge,” I know that I will be able to walk into the kitchen, open the door, and pull them out. I can simulate that sequence forward in my mind. An LLM would be able to generate words and would be able to describe how to get the tamales, but it won’t understand if you are talking about something completely new. With a human, we can walk them through and teach them new things, such as what a gas station is or what it is like to work in a kitchen. To learn new things quickly, AI needs to efficiently build new models, without requiring new nuclear power plants.

RNNs to attention to transformers to conversations

Claude: In discussing GANs, you mentioned the challenges of applying them to language due to its discrete nature. How do you view the path that language AI took instead - through transformers and large language models?

Jonathan: It’s been wonderful. We’ve seen three technologies come together, leading to chat AI such as ChatGPT and Claude. The three technologies are language modeling, transformers, and instruction tuning.

Language modeling is predicting the next word. We started with recurrent neural networks (RNNs), and those got even better when we started adding attention to focus on the particular words in the prompt that were most relevant to generate the next word. Transformers came about when we decided that attention was all we needed, and they were a significant improvement in language generation capability. Finally, instruction tuning converted good language models into useful conversationalists.

Instruction tuning has two parts. The first part is training an evaluation model, which judges generated outputs. To train this evaluation model, we have humans manually rate LLM outputs on how helpful they are, and then we train the evaluation model based on that rating data. The second part of instruction tuning is to use the evaluation model to guide reinforcement learning to tune an existing LLM. Since this part is autonomous using the learned evaluation model, it can do a good job of covering the space of possible things it could be asked, and since the evaluation model was trained on human ratings, it does a good job of steering the LLM model to generate useful text.

Surfing on abstractions

Claude: Your example of "lefty loosey, righty tighty" illustrated how an AI companion would need to understand and adapt to individual users' ways of thinking. How do you see the challenge of personalization in current AI systems?

Jonathan: I see the end goal of personalization is having the AI learn which abstractions you use to see the world. We understand by integrating abstractions on our mental stage. For example, I think of electricity as balls moving through a tube. This abstraction (metaphor) works for my purposes because if you cut the tube (the wire) the electricity doesn’t flow.

I see human thought as a process of surfing along abstractions. If you tell me the battery is dead, my mind thinks that it either ran out of little balls or can no longer put force on those balls. So to best teach us, teachers have to know what abstractions we think in, so they can teach us new abstractions building on those existing ones. That is personalized learning.

Conversation is our richest activity

Claude: The dialogue management systems you described (like Ravenclaw) were very structured. What are your thoughts on how modern systems handle conversation flow? Have we lost anything important by moving away from explicit dialogue management?

Jonathan: LLMs do a great job with dialog management by taking in the history of the conversation as part of their prompt, but as we mentioned on question 2, rich conversation requires a sense of personhood. Conversation is a unique medium. When you have a conversation with someone, you direct their mental scene and they yours. Movies, TV, and books are one way—they don’t do that. This cooperative aspect is what allows conversations to get to the heart of what is interesting in life. The best conversations teach.

Having conversations is one of the most sophisticated things we do as humans. With LLMs, we now have the opportunity to scale conversations. Encoding an AI to have a good conversation requires more than reinforcement learning over interpolated text generations, as is done with instruction tuning; and it requires more than predefined structures of old-fashioned dialog systems. Good conversations will require that the AI have goals and be able to plan to reach them. That goal can be implemented as a drive to teach and a drive to learn. Their drive for learning will make them a person in our eyes and it will make it worth it to spend the effort to teach them during conversations, and their drive to teach will directly benefit us and structure the conversation.

Understanding and humor

Claude: You mentioned working on helping computers understand sarcasm and quantity. How do you view current LLMs' handling of pragmatics and conversational implications?

Jonathan: One form of humor is you say something that isn’t what the listener expects but after a pause makes sense in an unexpected way. Consider the old joke about why you don’t see elephants hiding in trees. They are really good at it. The punchline makes sense in an unexpected way—since they are good at it, we don’t usually see them up there. To do humor, an AI needs a model of the audience to know both that the punchline will not immediately occur to them, and two that the punchline will make sense.

Sarcasm works similarly. To get sarcasm, you have to currently infer that what the person is saying is unlikely (you need a model to tell you that), and then you look for an alternative explanation of what they are saying, which could be the opposite. Imagine that someone says, “Oh, I love carrying rocks, especially when they are covered in fire ants.” You need a model of the person (or humans and fire ants in general) to know that this is unlikely to be literally true.

Great conversations and great relationships build when you learn models specific to a person. Most humans don’t like fire ants, but you have to get to know a person to know how that person’s preferences differ from the norm, and then you will get all of their sarcasm.

My talk at Data Day Texas 2025, What Superintelligence Will Look Like, will discuss what it will take for robots to use neural networks to learn the abstractions necessary to quickly learn the models necessary to build relationships and to have great conversations.

In its 15th year, Data Day Texas is possibly the longest running NoSQL, Big Data, Data....... conference in the world. Often referred to as a "Party masquerading as a data conference", its 2025 edition will be the best ever. Join us in Austin!

Special thanks to Jonathan Mugan for participating. Follow him on Linkedin.