Generative AI and Games

For the last year I've been building games and supporting technologies using generative AI--and I've tried to document what I've learned. I've also spoken to a dozen leaders at game studios and technology companies who are powering the generative AI revolution.

Read on for a summary of the most relevant articles and conversations I've had in the last 3 months:

Semantic Programming and Software 2.0

This is the article that covers the programming methods that allowed my team to win the LA AI TechWeek Virtual Worlds hackathon, building a playable RPG in 1 day. We used Anthropic 's Claude LLM (with 100K context), Blockade Labs , Scenario , Beamable and Unity as the key technologies. Learn how we did it:

Open Source Experiments in Generative Gaming

Some of the earlier experiments--which we've open-sourced--by myself and my team at Beamable. This demonstrates the use of GPT into the development and real-time operation of games, as well as use of generative images from Scenario:

75 Lessons from Learned from Live Game Development

This isn't strictly generative AI, but it is my own hard-won lessons from building and shipping games to over 20 million players--as well as some of the failures and learnings I collected along the way. If generative AI can accelerate development velocity, it supports one of the most important pillars of my thesis: more "shots on goal" to get your game right for your audience.

Digital Identity and the Evolution of Creativity

I presented at the MIT Media Lab 's Metaverse course, discussing how virtual worlds started as a place to manifest our digital identity--and are now a platform for our creativity... and soon, our will. The latter is enabled by generative AI agents.

Building the Metaverse

On this season of Building the Metaverse, I've been speaking with builders and leaders using generative AI to build their business. There's a focus on games and virtual worlds, but many of these technologies could be applied to a wide range of use cases.

I provide direct links to the YouTube conversations below, but if you click on the titles, you'll go to a summary article that contains supporting materials and show notes.

Text-to-World

My conversation with Kayla Comalli was about her startup that is using a text prompt to spawn an entire world to explore. We took a look at her product and the supporting technologies like Sloyd, Polyhive and Unreal Engine 5.

Creative Technology and Generative AI with Bilawal Sidhu

Bilawal Sidhu is doing some of the most interesting creative work with generative AI, neural radiance fields (NERFs) and photogrammetry. Some of his examples have been widely shared--and he has over 100M downloads of his augmented reality filters for TikTok. Learn about his work and methods in our conversation:

Generative AI and Voice Synthesis with Zach Johnson

I recorded a conversation with Zach Johnson about how generative AI will change the way we incorporate voices into games and simulations. We also show off some amazing demos of using Zach's voice synthesis technology inside a flight simulator. We also talk about the nature of art, and the evolving creator economy around AI.

Generative Emotes for Avatars

Expression through avatars is one of the most complex frontiers for generative AI: it covers everything from modeling to rigging to animation. In my conversation with Yassine Tahi , we discussed how Kinetix is bringing together all of these elements and simplifying access through text prompting and a creator economy.

AI Storytelling and Narrative with Linus Ekenstam

Linus Ekenstam is creating a storytelling product using generative AI. We talked about how artificial intelligence is changing product design--and civilization itself.

ChatGPT and Game Development with Cameron Wills

Cameron Wills created the most-downloaded AI tool on the Unity Asset Store. We discussed his journey as an indie game developer, and what led him to make a generative AI tool to speed development:

AI Art, Creativity & Copyright with Kris Kashtanova

Kris Kashtanova obtained the first copyright for an AI-generated work, her comic book: Zarya of the Dawn. We discussed her journey from photographer to AI artist, and what she's learned from her quest to obtain copyright on her work.

Generative Graphics Workflow for Games with Jussi Kemppainen

Jussi-Petteri Kemppainen is building an indie game using generative AI. He's created--and documented--a workflow involving tools that take art from Midjourney, creates projection maps, sets them up in Blender and then into Unity. Watch this video if you're an artist or game developer who wants to learn how to get generative concept art right into your game engine.

Virtual Beings and Edward Saatchi

Inspired by the Culture Series by Iain Banks, Edward S. 's vision is to create a virtual society with millions of AI-powered virtual beings. We discuss the importance of created simulated environment in which virtual beings take actions--not simply engage in conversations.

Generative Art Assets for Games: Emmanuel de Maistre

Scenario is creating generative AI tools that let you create game assets by training your own generators--and text-prompting individual assets into existence. We discuss how Scenario works, Emmanuel de Maistre 's vision for the future of game development, and his entrepreneurial journey.

Generative Games with Language Models: Hilary Mason

Hilary Mason leads Hidden Door, a startup building multiplayer games built around large language models (LLMs). Before Hidden Door, Hilary led machine learning for Cloudera--and was previously a professor of computer science. We discussed how language models will enable whole new categories of games. Some of the language models that may be put into practice may be simpler, CPU-driven models rather than the ChatGPT-style huge models you might be acquainted with. Learn more: