The Only Open Source AR Glasses Powered by AI

After the tumultous release of the Humane AI pin and the Rabbit R1 the space of AI wearables heated up. In this issue we will dive into this topic with Bobak T. the CEO of Brilliant Labs , who is building Frame: the first OPEN SOURCE AR glasses powered by AI. And since we are talking about open source, there is no better platform than Hugging Face to take the Product Spotlight section of today’s issue… with a 3D twist of course ??.

You can subscribe to and support XR AI spotlight right on Substack , with new issues appearing directly in your inbox, plus bonus issues only available to our supporters.

If you’d like to sponsor this newsletter, and get your name in front of an engaged audience of professionals and creatives, just contact me here on LinkedIn ?? Gabriele Romagnoli.

Interview with Bobak Tavangar

Can you give us an impression of how a day in the life of a typical Frame user looks like?

Bobak Tavangar: Right now our early customer base is largely developers. We have developers from various industries like healthcare, education, and government building all kinds of interesting applications. For example, a doctor or nurse might use it for patient care, a teacher or student for educational purposes, or a marine biologist for field research. It really depends on the specific domain and what folks are doing. We're still in the early stages, and it's going to take time before these devices conquer the mass market.

What challenges have you faced in developing and launching Frame?

Bobak Tavangar: Ensuring the hardware is lightweight, affordable, and has decent battery life while integrating advanced AI capabilities has been a significant technical challenge. Additionally, fostering an open-source ecosystem that encourages innovation while maintaining competitiveness requires careful strategy and execution. Despite these challenges, the positive response from the developer community and early users has been very encouraging.

How do Frame glasses communicate back to the user?

Bobak Tavangar: Frame uses a display optic in front of the eyes and audio through connected devices like AirPods. The display optic provides a heads-up display with text, graphics, emojis, and images, while the audio component, handled by your existing headphones, allows our AI assistant, Noah, to respond to you. Developers have a lot of flexibility in customizing the visual and audio output to create immersive and interactive experiences for users.

What makes Frame stand out in the crowded AR glasses market?

Bobak Tavangar: One thing that sets Frame apart is that it is completely open source. This allows developers to inspect, verify, and modify both the hardware and software. This openness unlocks innovation and customization, and allows anybody to create unique applications across various industries. Additionally, our focus on affordability and a lightweight design makes Frame accessible to a wider audience, encouraging more creative uses and applications.

How does Noah, the AI assistant, work with Frame glasses?

Bobak Tavangar: Noah is our AI assistant integrated with Frame. It uses a base model, currently GPT-4, to understand the intention behind user inputs, whether it's audio, text, or images. Noah then performs function calling to select the appropriate tool for the task, such as image generation with Stable Diffusion, precise web searches with Perplexity, or reasoning over information. This modular approach allows developers to add new models and customize how Noah responds to specific queries.

Can you explain the role of open source in Brilliant Labs' strategy?

Bobak Tavangar: Open source is foundational for us for two main reasons. First, it's important that these new, world-changing technologies are open, inspectable, verifiable, and modifiable. This ensures transparency and adaptability. Second, strategically, we believe that in an era where developers and customers seek more reciprocity from platform providers, an open ecosystem will foster innovation and build a strong community around our products.

What are some examples of applications developers have built on Frame?

Bobak Tavangar: Developers have built a wide range of applications on Frame. For instance, some are working on face recognition and retail-oriented applications, while others focus on medical use cases like recognizing symptoms and providing step-by-step instructions. There are also applications for translation in niche circumstances and various educational tools. Our vibrant Discord community shares these projects, and it's exciting to see the diverse and innovative ways developers are using Frame.

How do you ensure privacy and trust with the data collected by Frame?

Bobak Tavangar: Privacy and trust are paramount for us. We approach data management like a bank, acting as custodians. Users have control over their data, stored on a server with keys only they can access. They decide who has access to their data and can edit or remove it as they see fit. We believe this model is essential for building trust and ensuring users feel secure using our technology.

What are the current business models for Frame and Brilliant Labs?

Bobak Tavangar: Our business model includes a one-off hardware sale and subscriptions for pro-level access to AI services through Noah. Users can access more advanced models or higher usage tiers for a monthly fee. Additionally, we plan to offer AI personalization services, which require significant compute power, on a subscription basis. This approach allows us to provide ongoing value and support while generating recurring revenue.

How do you balance being open source with remaining competitive in the market?

Bobak Tavangar: Balancing open source with competitiveness involves focusing on areas where we can add unique value, like creating lightweight, affordable hardware with decent battery life. We don't try to compete directly with companies specializing in model development but rather focus on integrating and processing data to make interactions with AI increasingly personal. Privacy and trust are key differentiators for us, ensuring that users feel secure while using our technology.

What is your impression of how social media and influencers reacted to the reviews of the Rabbit R1 and the Humane AI Pin?

Bobak Tavangar: It's definitely interesting. On one hand, people say they love innovation and want to celebrate startup culture, but on the other hand, the internet has become like the old coliseums of Roman times. There's a lot of blood lust, and people seem to get a dopamine kick from tearing down others. This reaction is often driven by the business model of the internet, which rewards clicks and views. While there are some trusted publications, a lot of the reviews are more about generating views than providing thoughtful analysis.

How do you see the market for wearable AI devices evolving in the next three years?

Bobak Tavangar: In the next three years, we expect foundation models to become smarter and more capable, with reduced latency as the cost of GPUs comes down and more models run locally on devices. AI assistants will become more interactive and capable of performing complex tasks both digitally and in the physical world. As IoT embeds itself further, these agents will act on various devices around us, making our interactions with technology more seamless and powerful.

Check out the full interview right here ??

Product Spotlight: Hugging Face 3D Goodness

Hugging Face is the leading platform and community hub for AI, particularly focused on natural language processing and machine learning. It serves as an open-source repository for AI models, datasets, and tools, and makes it possible to play around and test some of the latest applications of AI even when it comes to 3D!

By clicking on this link you will be directed towards all the 3D-related models.

You can filter and sort all the entries and you can run each one directly in your browser.

Here are some examples that stood up for me:

• Zero-shot depth estimation : predict the depth of an image and then uses 3D Point Cloud to create a 3D object.
• Roblox 3D assets generator : generate high poly mesh for your Roblox game (comes with a full tutorial )
• Depth Anything : add depths to any 2D image
• 3D Room Layout estimation : It uses the Geometry-Aware Transformer Network to predict the 3d room layout of an rgb panorama.
• SF3D : a state-of-the-art method for 3D mesh reconstruction from a single image

That’s it for today, and don’t forget to subscribe to the newsletter if you find this interesting

See you next week