Sensing: The Next Frontier for ChatGPTs of the World?

ChatGPT and similar tools built using large language models (LLMs) have revolutionized how we interact with AI. Yet, they still face notable limitations. For instance, I once asked ChatGPT to write a children's book titled Bunny Diaries, inspired by Owl Diaries, for my 9-year-old daughter. Her verdict? “It made me cringe!”.?

Similarly, the infamous "human hand problem" in LLM-generated images and videos highlights subtle yet critical visual inaccuracies. While advancements in training methods and cost functions are likely to mitigate such issues, they won't address a deeper, more fundamental challenge: the ability to sense.

Beyond Pattern Matching: The Power of Sensing

What is “sensing”? While the term is often associated with physical perception, here it means the dynamic process of interpreting and making sense of data streams in real time. The key word here is “dynamic”.

LLMs are trained to predict the next element in a sequence—text, speech, or video—based on patterns detected in prior data. This method is powerful but passive. What’s missing is an essential human capability: dynamic sensing.

Let’s consider sensing in familiar contexts like hearing and vision. Humans excel at interpreting sensory input through a feedback loop between perception and cognition. Two scenarios illustrate this:

• Noisy Speech: If you hear garbled speech and replay it, the second attempt often sounds clearer—not because the audio changed but because your brain processes it differently, using context to adjust perception dynamically.
• Seeing What You Desire: When you decide to buy a black car, suddenly you will notice more black cars on the street than before. This isn't because more black cars appeared, but because your perception has shifted.?
• Finding Waldo: Once you find Waldo in an image, you can’t “unsee” him. Your brain locks onto that visual cue, and it becomes a fixed point of recognition.

This interaction between thought and perception helps humans refine their understanding, interpreting the world through context and adaptability.

Bridging the Gap: Teaching LLMs to Sense

Sure, we can—and likely will—train LLMs differently in the future to be more inquisitive and improve their knowledge bases dynamically. But this is not where LLMs are today.

However, by leveraging creative prompting alongside computer vision and AI-based audio processing stacks, we can develop LLM-based systems that emulate sensory capabilities.This involves? involves using well-designed prompts to enable LLMs to provide iterative feedback to the AI subsystems they rely on for interacting with the real world, For instance:?

• Vision Systems: Much like how our eyes adjust to focus on a nearby object to adapt to bright sunlight, a camara could fine-tuns its focus and saturation in real-time based on the LLM′s analysis of what is most relevant in these scenes. For example, if the system detects? an important object in motion, it could sharpen its focus to capture that detail.
• Audio Systems: Similar to how our brain filters out background chatter to concentrate on a single conversation at a noisy party, a microphone pipeline could adjust to suppress unwanted sounds. For instance, it might zero in on the voice of a specific speaker of interest in a crowded room, enhancing clarity and focus for the LLM.

These enhancements would enable LLMs to process sensory data iteratively, bridging the gap between static data interpretation and dynamic sensing.

The Path Forward

The technology to build adaptable subsystems already exists, and fields like prompt engineering are well-developed. However, the challenge lies in integrating these components into cohesive systems that allow LLMs to engage dynamically with the world.

Each subsystem—be it audio, visual, or contextual—must be carefully designed and built. These components must then be harmonized to create a synergistic interaction between LLMs and their environments.

By moving beyond passive pattern matching, we can transform LLMs into tools capable of sensing and engaging dynamically with their surroundings. This advancement would significantly expand their utility, enabling them to achieve far more than just searching for cat videos on the internet.

How Far is This Future?

All the components of this future are within reach for AI experts. The team of https://AZcare.AI has already taken a bold step forward by developing the industry′s first sensing agent to serve as a personal health agent. This milestone signals the vast potential ahead for AI systems that sense and adapt, transforming industries and redefining how we interact with technology. And many are following in their footsteps soon.