10 years of AI-specialized chips

While Google’s digital services — spanning Search, Cloud, YouTube, Maps, Workspace, Gemini, and so much more — are some of the most popular in the world, it’s safe to say they would never have gotten that way if it weren’t for the hardware they were built on. From our earliest days stringing together warehouses full of servers, we have focused on the largest and smallest bits of our infrastructure.?

The latter, in the form of custom silicon chips, is playing an especially important role these days, as we seek to optimize the performance and sustainability of our AI and computing services. In this edition, we take you inside a decade of TPUs, our AI specialized chips, and delve into the future of Axion, our recently released Arm-based CPUs.

TPU transformation

Just over a decade ago, a group of Googlers discovered that the company’s AI compute demand was going to outpace our infrastructure at the time. The discovery came as research teams began thinking seriously about launching speech recognition features at Google’s global scale.

“We did some back-of-the-napkin math looking at how much compute it would take to handle hundreds of millions of people talking to Google for just three minutes a day," Jeff Dean, Google's Chief Scientist, said in an interview. "In today's framing, that seems like nothing. But at the time, we soon realized it would take basically all the compute power that Google had deployed. Put another way, we'd need to double the number of computers in Google data centers to support these new features.”

“We thought surely there must be a better way.”

The team looked at different approaches that existed on the market, but ultimately realized they were not able to meet the sheer demand of even those basic machine learning workloads our products were operating — let alone what might follow in the years to come.

Google's leader realized we were going to need a whole new kind of chip. So, a team that had already been exploring custom silicon designs enlisted Googlers from other machine-learning teams and laid down the framework for what would ultimately be our first Tensor Processing Unit , or TPU.

A single, specific purpose

Where Central Processing Units (CPUs) are designed as the jack-of-all-trades general-purpose “brains” for a computer, and GPUs, at the time, were specialized chips designed to work in tandem with a CPU to accelerate complex tasks in graphics, video rendering, and simulations, TPUs were purpose-built specifically for AI. TPUs are an application-specific integrated circuit (ASIC), a chip designed for a single, specific purpose: running the unique matrix and vector-based mathematics that’s needed for building and running AI models.

Our first such chip, TPU v1, was deployed internally in 2015 and was instantly a hit across different parts of Google.

“We thought we'd maybe build under 10,000 of them,” said Andy Swing, principal engineer on our machine learning hardware systems. “We ended up building over 100,000 to support all kinds of great stuff including Ads, Search, speech projects, AlphaGo, and even some self-driving car stuff.”

In the decade since, TPUs have advanced in performance and efficiency across generations and spread to serve as the backbone for AI across nearly all of Google’s products.

Continue reading on Transform with Google Cloud.

Why Google keeps building custom silicon

Everything runs on CPUs.

Compute power, delivered by a range of chips that include CPUs, GPUs, and TPUs, underpins nearly every large-scale service in the cloud.

GPUs and our own AI-optimized TPUs have grabbed most of the attention lately, for their role in accelerating progress in the AI era . And yet it’s those general-purpose CPUs that still handle the lion's share of workloads — everything from computationally-heavy data analytics and financial modeling applications to more straightforward web applications or little-used but important microservices.

The world seems to agree just how important CPUs are, given the response to our announcement of the Google Axion Processor at Google Cloud Next ‘24 . The unveiling of Axion as our first custom Arm-based CPU quickly became one of our most widely reported and discussed releases this year, and we’ve seen considerable interest from customers since then.

As a company with its roots online, Google has always prioritized computing hardware, going back to the early days of engineers stringing up servers in garages and industrial spaces around the Valley. In fact, this warehouse-scale computing effectively laid the foundation for what would become Google Cloud. Our push into chip design came more than a decade ago, but the rationale has always been the same: the more we could do to shape our own hardware and software systems, the more we could do to shape our own destiny.

Axion is the latest leap in this journey — though far from the last, as we followed it up a month later with the introduction of the sixth generation of TPUs, named Trillium . To continue innovating in technology, we’ll continue innovating in silicon.

It was our experience of building such specialized chips not only for AI but also mobile and video streaming that gave us the confidence to tackle the more generalized though complex needs of CPUs (it sounds counterintuitive, but a general-purpose chip like Axion needs to handle a wider range of applications, which necessitates its more complex design). We now knew how to bring together teams of software engineers, hardware engineers, researchers, and partners under one roof to co-design custom silicon from the ground up.

In many ways, the release of Axion brings our work on collaborative, capabilities-first, customer-focused hardware design full circle. What started with the dream of making the world's information more accessible — anytime and anywhere — from a few server racks in a California garage is now the basis for what draws customers to Google Cloud. Axion is the latest milestone of that decades-long focus.

Continue reading on Transform with Google Cloud.