From Perforce to Piper: The Evolution of Google's Source Control System

In the fast-paced world of technology, scaling up is often more complex than it seems. In 2011, Dan Bloch, the tech lead of the Perforce admin team at Google, published a paper titled "Still All On One Server: Perforce at Scale." This paper chronicled the journey of Google’s source control system, which astonishingly serviced "over twelve thousand users" daily, all running off a single Perforce server housed under a stairwell in Building 43 on Google's main campus.

The Perforce Era: A Single Server for a Giant

By 2011, this server had been operational for eleven years, from Google’s startup days to its status as a public company. It had processed a staggering PR #20,000,000 and was executing "11-12 million commands" daily. Despite some successful scaling efforts, the server faced increasing challenges. TCP connection failures and maxed-out CPU usage became common, prompting a dedicated team of eight admins to keep it running smoothly.

Google had long recognized the risks of relying on this single server. As one of the largest repositories in any source control system globally, alternatives were scarce. This predicament mirrored Linus Torvalds’ creation of Git in 2005, designed to handle the Linux kernel repository's immense size. By 2014, Google’s monorepo had grown to a point where it changed "approximately 15 million lines of code in approximately 250,000 files" weekly — a volume equivalent to rewriting the Linux kernel from scratch every week.

The Birth of Piper: An Innovative Solution

By 2008, Google engineers were exploring alternatives to their overburdened Perforce server. The idea of breaking up the monolith was considered but ultimately rejected, a decision that would influence code complexity management for years to come. Google’s commitment to the monorepo approach was solidified in its 2016 paper, "Why Google Stores Billions of Lines of Code in a Single Repository."

While SVN (Subversion) was briefly considered, it soon became clear that a new system was needed. Thus, Piper was born. Named after engineers' love of planes and short for "Piper is Piper expanded recursively," Piper was to be a distributed system built on Google’s infrastructure, originally Bigtable.

The Long Road to Migration

The transition from Perforce to Piper was no small feat. It took over four years to migrate Google’s entire monorepo to Piper. Initially, there were 300 tools relying on the Perforce API, with production dependencies cropping up unexpectedly. The team had to ensure the migration didn’t disrupt Google’s end-user experience, a complex task given the intertwined nature of the systems.

Adding to the challenge was the Oracle lawsuit over Google's use of licensed Java API interfaces in Android. This legal battle, which lasted until 2021, raised concerns about seamlessly migrating without copying the Perforce interface. Engineers employed clean room design, where specifications were written by one team and implemented by another with no knowledge of the original API, to navigate this issue.

The High-Stakes Cutover

The migration project, initially a cool and exciting idea, took on new urgency as time passed. The load on Perforce continued to rise, increasing the stakes. Notable systems like Blaze (Google’s build system, later open-sourced as Bazel) and TAP (Google’s testing platform) were developed on top of Perforce APIs, making the migration even more critical.

By the time the migration reached its final stages, commits were being double-written to both Perforce and Piper. The Piper team, consisting of ten engineers, prepared meticulously for the cutover, aware that any mistake could potentially take Google down.

On the day of the migration, tension was high as the source control system became read-only while state was transferred from Perforce to Piper. After a few tense moments, the migration was complete, with no loss of state and Google’s production instances remaining unaffected. The years-long effort had paid off.

The Aftermath and New Horizons

The successful cutover to Piper immediately reduced Google’s operational risk and paved the way for new systems, such as Tricorder, Google’s static analysis tool. Post-migration, the number of automated commits surged, reflecting the newfound efficiency of the system.

The Piper migration is a testament to Google's innovative spirit and engineering prowess. It showcases a time of daring decisions and scrappy problem-solving, even eight years after Google’s IPO. Today, Google's internal tools are seen as state-of-the-art, but the journey to Piper reminds us of the challenges and triumphs that pave the way for such advancements.

Final Thoughts

Google's migration from Perforce to Piper is a remarkable story of resilience, innovation, and forward-thinking. It highlights the importance of daring to make bold decisions and the meticulous effort required to see them through. As technology continues to evolve, the lessons from this migration will undoubtedly inspire future generations of engineers.