Amazing Excerpts from Paige Bailey, Google, on GenAI (ETLS Vegas 2024)

It was such a pleasure to have Paige Bailey , former lead product manager for the Code AI, PaLM2, and Gemini generative AI models, speak at ETLS Vegas in August. She is now AI Developer Relations Lead.

In August, she shared her journey that started at Google DeepMind, and her journey that led to Gemini.

One of the coolest things she hinted at was how Gemini is replacing many custom AI models that Googlers built to power so many of the legendary Google properties, including Search, YouTube, Sheets, Docs, and more...

She talked about how Google has built so many AI capabilities over the last decade, which until recently, only Googlers got to see and use.

(Apparently, the release of ChatGPT changed all that. From my interaction with some Googlers, it appears this has led to the largest mobilization within Google in ten years, with the rise of Google Cloud.)

She talked about how Gemini Flash is powering so many Google product and experiences, because of its speed and power-efficiency (i.e., cheap to run) — this is absolutely required when you're serving billions of users daily.

"Gemini 1.5 Flash model built a new quadrant in the model performance space, because it's so cost efficient without sacrificing performance."

These are the models that are showing up in the Chrome browser, as well as Pixel phones — all data is kept locally, with no data being sent to a server somewhere.

(I see in the Slack channel in the side, during her talk, I noted that Gemini had just dropped prices by 75%. I love the phrase, "the cost of intelligence is rapidly heading to zero.")

This is the part of her talk that I found so exciting — she described an internal Google initiative called "Code AI", which has been trained on over 25 years of engineering data. That's 500K years of aggregate engineering activity of Googlers, "some of the brainiest people I've ever met."

It's astonishing how wholly incomplete our view of software engineering is, compared to what she describes. It's so much more than code, test, deploy, run.

Bailey: "Over the last 25 years, we've been capturing telemetry on everything that our developers and engineers do internally.

"If you encounter a bug, we know everything that you need to do in order to resolve it. We know all of the logs associated, we know all of the compute impact for running a piece of code. We know if you tried out a couple of different APIs and then settled on a different one.

"That's just in your IDE. Outside of it, we know if you looked up documentation, if you asked a chat message, if you got a response back from a Q&A system. If you went to the micro kitchen to get a snack, we know how every single one of those aspects of the developer workflow impacts that final product."

"So if you want to build an AI system that understands tools use, function calling, how to select different APIs, or how to debug different aspects of a software engineering workflow, we have that fully detailed telemetry."

Wow.

- 500,000 aggregate years of software engineering activity

- over a trillion tokens

- over 80 million high-quality code review edits

- 90% + Google engineers are using our converged dev tools stack means Al "just works"

To me, it's amazing how big the "whole" of "coding" is — and the value when you've captured all of it. Telemetry, issues, service calls, code commits, compute, etc...

Some mind-blowing stats:

- 26% of all code at Google is generated by machine learning, and in some weeks, 50% of checked-in code is generated by machine learning

- Code AI is accelerating code review by automatically applying reviewer feedback, eliminating the need for manual incorporation — automatic one-click merging of proposed changes from the model

- AI-powered code performance optimization can lead to significant cost savings, with single code changes in C++ or Python potentially saving tens or hundreds of millions of dollars at Google scale

Super cool demonstration of AI-assisted migration between libraries.

This is such a killer application of the huge Gemini 2MM token context window from Bailey.

"You can also generate detailed friction logs from user videos. User experience researchers often might sit with a user, ask them to test out a feature, record the entire painful process, and then afterwards meticulously document 30 minutes, an hour of video content of what a user is doing, where they ran into trouble. Their goal is to learn where to create better documentation or better product features in order to ease that pain over time.

"Gemini is able to do this out of the box. It can generate detailed friction logs from user videos. All you have to do is say, Hey, here's the video. Generate a detailed friction log. and then give me a summary of which product features we might implement and prioritize them. Think about how much this could empower the entire UX space, it's huge."

"These studies typically take months. Imagine if we could automatically process thousands of user sessions, cluster them by usage patterns, the problems they're running into.

"This is as opposed to making decisions on the small user sample we chose."

Amazing and mind-expanding!

Bailey: "As a former product person, I love PM work. I think it's very important. I am back on the engineering ladder, which I also am very excited about. But when I was a product lead for frameworks and for APIs, you often had to summarize large swaths of user information, synthesize multiple support tickets, etc.

"At Google Scale, this ends up being on the order of tens of thousands, hundreds of thousands of pieces of feedback across discourse forums, stack overflow, GitHub issues, support tickets, perhaps pieces of feedback that people have given on social media.

"It's really challenging to be able to read through that all yourself. It's a herculean task, pretty much impossible.

"You're stuck in this weird place where you do ad hoc PM work, pulling in six customers, hoping that they're a representative sample and implementing their feature requests as opposed to getting a full analysis of all of your tens of thousands, hundreds of thousands, millions, or billions of users."

Bailey: "We're rolling out this capability that we're using inside of Google to everyone. It's not just within the context of this IDE, it's everything. It's Drive, its Docs, its Sheets, the information that you might have in an Meet conversation, all of the telemetry that we're capturing for all of the DevOps tooling that you're using on Google Cloud.

"Software engineering is a team sport. It's cross-disciplinary, so it's software engineering but also multiple other functions. We are multifaceted humans.

"If anybody in the audience is a startup working in the AI space, we have a really compelling startup program. Please contact me afterwards if you have any interest in joining. We have a trusted tester program for the newest versions of our Gemini models.

"I also want to close by saying this entire process is really magical to me because it reduces the friction from many of us having an idea to actually getting it out into the world in a production capacity.

"There have been so many instances where there are video games that I wanted to create or VS Code extensions or Chrome extensions that I was able to get perhaps 85% of the way there, but then not that last 15%.

"Hopefully this tooling not only gives us the ability to generate the code, but also to deploy it, maintain it, make it production ready and make it secure. Thank you so much. And go try out Gemini 1.5 Pro and 1.5 Flash at https://aistudio.google.com . Tell us where it works."

What a wonderful talk—thank you so much, Paige! It was great to re-watch this after seeing the Dr. Jeff Dean interview I wrote about here:

https://x.com/realgenekim/st/RealGeneKim/status/1845365233625260345

You can watch Paige’s complete talk here (just register with your email address):

https://videos.itrevolution.com/watch/1002959711

I’m thrilled she’ll be joining me for a Q&A at ETLS Connect next week—hope to see you all there!

Details and registration here: https://itrevolution.com/product/etls-connect-october-2024/