Audio-Visual Technology at the core of AI-Powered Smart Buildings: Great ideas from a whodunit SciFi series.

Science fiction often inspires technological innovation, providing a glimpse into the future. The genre's creators often speculate on emerging trends and possibilities, inspiring inventors and engineers to turn fiction into reality. This concept is evident in the emergence of AI-powered smart buildings, as seen in the TV show "A Murder at the End of the World."

The show's characters interact with advanced AI technology seamlessly integrated into a smart hotel. The hotel's AI system is accessible without phones or tablets, allowing guests to interface directly with the built environment. This concept raises exciting questions about how we will interact with AI in the near future and how we can communicate with our surroundings without traditional devices. And at its heart is AV technology.

Watch this clip- and then I'll break down the technology.

To achieve this level of interaction, several critical components are needed:

Ubiquitous Microphone Arrays

Microphones must be strategically placed throughout the building to capture user requests anywhere within the environment. This infrastructure enables the AI to respond to commands without needing Apps or devices. Microphones can also locate the user in the room- to give the AI context.

Computer Vision Integration

Computer vision technology provides the AI system with additional context. By tracking wearables, headsets, and user positions within the room, the AI system can anticipate user needs and provide personalised experiences. When overlaying data onto the physical world (via projection or augmented reality glasses)- understanding the user's location, orientation, and even eye movements is essential to make the experience seamless.

Advanced Audio Playback Systems

Advanced audio playback systems are required to simulate a voice coming from a specific point in the room. These systems, including beam steering arrays and automated mixing via a centralised DSP- help create a more immersive and engaging experience. In the example clip, she starts the interaction as audio-only and then transitions into wearable glasses, where she can see the AI as a person. She was already looking in the right direction as the sound came from a fixed point in the room- as if the AI assistant was standing there the whole time.

Projection Capabilities

A projection system must be integrated into the building's design to present information visually. This could be achieved using traditional projectors (with projection mapping) or more advanced holographic projection technology. Presenting information anywhere within the room enhances the overall user experience. Projections allow the experience to overlay on the real world- not fixed to a screen or the user's device.

Integration into Building Control

To control the building's systems effectively, the AI must integrate with building systems via APIs. This integration enables the AI to manage functions like lighting, temperature, and security systems, creating a fully immersive smart building experience. All your components must be on the network- and select vendors with open, well-documented APIs.

With the AI's ability to understand the functions of the APIs, you do not need to specifically program actions. Instead, the AI will take requests, put them into context, and determine the best function to call from the available APIs. This is a giant leap away from the traditional approach of hard-coding App functions or logic workflows. The AI has the ability to work out what to do without the limitations of fixed programming. You just need an integration platform as an abstraction layer to normalise data and communication. This is what we do at PlaceOS.

Augmented Reality Glasses

While not strictly necessary for the system's core functionality, augmented reality glasses could significantly enhance the user experience. These glasses could provide additional information and interaction, offering users a seamless way to engage with the AI system and the smart building's features. There needs to be a seamless handover (like in the example clip) between AV and Glasses. There are two apparent technologies to point out how close we are to the reality of the clip: Meta/Ray Ban have ChatGPT built into their glasses, which allows you to talk to ChatGPT and hear responses via bone-conduction audio. And Apple is launching into spatial computing with Vision Pro. The race is on, but we must consider the whole design, including the transfer and handover between personal devices and installed AV.

AV is your secret weapon.

I've spoken publicly about how AV professionals can transfer knowledge to IoT and smart buildings. And now I am seeing it come the other way around. AV is no longer the end-point device- it's the interface.

Here are some technologies you could use today to build a smart building offering like the example A Murder at the End of the World clip:

• Conversational AI: There is ChatGPT/Open AI (see our integration docs on how to get it working with PlaceOS). But my favourite example of excellent conversation skills is Pi AI. I interviewed Pi on my Podcast- it was weird!
• Ubiquitous audio-visual. Any input (microphone) to Any Output (speaker) at any location on the network. This is precisely what fellow Australian company Audinate provides with Dante. This becomes the ears and the voice of the AI system. Furthermore, Dante now does video so that we can send visualisations from the AI anywhere in the building.
• Automation Platform so the AI can trigger actions: PlaceOS- that's us.
• Augmented reality glasses (this YouTuber does a great overview of the current state of things). We could get close to the example clip- but not in a low-cost- low-form factor option. This will change rapidly.

We can create a future where buildings are intelligent and responsive to our needs, offering a more immersive and engaging experience. Combining the technologies above could make this smart hotel example a reality today. Stop focusing on Apps, forms and buttons- let's get smart about our buildings.