Language Policy at Google: Let's Go!

Last week, I wrote a post that was largely about how Google decides whether to support a programming language for general use. I mentioned that languages developed by Google, like Go and Dart, and languages advocated by Google, like Kotlin, were special cases. Sameer Ajmani (who is the director responsible for Go) challenged me to write about Go.

It turns out that Sameer wrote about Go's early development at Google a couple of weeks ago, and covered much of what I would have said. So, I'll give it a bit of my own spin. I was there before Sameer joined the Go team, traveling a parallel path with Java. I'm an outsider to the Go team; what follows is my secondhand perspective, gleaned from the one-off meetings, lunches, and quarterly planning meetings I had with the Go team until I left that part of Google in 2018. I'm sure my recollections are rife with mistakes.

Largely for historical interest: the first conversation I had about Go was in a short meeting that I had with Rob Pike in 2007. He was developing what would become Go, and I had recently written Java's concurrency spec; my manager at the time, Brad Chen, thought it would be great if Rob and I talked. All I knew about it was that Rob (who worked down the hall from me) was locked in a room with Robert Griesemer and Ken Thompson doing something. The rumor was that Rob and company had gotten (rightly) frustrated at the experience of using C++ at scale, and had decided to fix the problem, using some of the abstractions that he had developed during his tenure at Bell Labs. I talked about my work on Java concurrency; he said they wouldn't have shared memory based concurrency in Go; I was somewhat surprised at that, and said so; we talked about CSP (which I had studied) and what became goroutines; and - effectively - that was the end of that.

I remember the big unveiling. We all crowded into Charlie's to watch Rob talk about what they had built. Go users will recognize what what we saw - a small, simple programming language that compiled quickly (they had spent a lot of time getting it to compile quickly, and were proud of it). It was a return to the simplicity of C, with modern programming constructs that allowed it to scale in a way that C couldn't.

Except that I just didn't get why we would want it at Google. We had a language that compiled quickly and was garbage collected - Java. And we weren't likely to replace C++'s performance sensitive use cases with a language that used GC (and didn't have an optimizing compiler yet). Go was simpler to learn, but it didn't seem like it had something that would displace one of the existing languages. In the parlance of my last post - what was the something critical it could do that the other languages didn't do?

I talked about the policy for programming language approval in my last post. This was before I got involved in it. In Go's case, there wasn't a lot of ceremony. Our VP at the time, Bill Coughran, simply approved it for use, and that was the end of it.

Or, rather, that was the beginning of it. Because the notion of a "supported language" isn't a magic wand that you wave that will get people to use it. The people who make decisions about which programming languages to use are a lot smarter than that. They asked the same questions that we were asking as part of the policy. What educational resources are available? How well does it integrate with existing Google technology? How do you test? What are the performance characteristics of programs written in this language? Why would I switch to another language when all of my engineers are happy and productive in the existing language?

You can read Sameer's post on what they were doing at this time for the insider's view on some of this.

Most importantly, in Go's case, regardless of the fact that the language had technically been approved, SREs refused to support services written in it. If you wanted to run a server with it, you were on your own. In addition to the dozens - many dozens - of monitoring and management tools and thick client libraries that had been built for Java and C++ that were missing for Go, the SREs were worried about cases where the engineers writing code in the existing languages had decided to simplify things for themselves at the expense of the people running the services.

For example, C++ developers had this tendency to deal with any problem they detected in their code by causing the service to crash. This was a nightmare for SREs, because any given code in the system could cause a series of restarts.

More pertinently for Go - Java GC had really bitten them. At the time, worst case GC could stop a service for multiple seconds. We had critical projects dependent on Java - what do you do when any of your services could stop for multiple seconds at a time? Do you restart them? Do you replicate all the traffic and hope you hit a live instance? You definitely had to overprovision memory. SREs had nightmares about this (and, on the Java team, we were doing our best to improve matters), and engineers simply didn't think about it.

It was Sameer who really solved these problems. If anyone out there wonders how you get to be in charge of Go, the answer is "talk with SRE leadership, get a list of about 80 major projects, and orchestrate a bunch of geniuses who want to go in their own direction hacking on programming languages to cross them all off, one by one". They had to do everything: from making sure that they could communicate with Google's storage backends to driving submillisecond GC latency to authoring load balancer APIs to building tools that could expose performance data and metrics compatibly with all of our production monitoring services. Under Sameer's leadership, the certifiably brilliant people on the Go team worked like demons, and accomplished something incredible - they turned Go into a programming language that you could really use for production services at Google. Pinocchio became a real boy. The SREs relented, agreeing to support Go services in 2013.

Something else happened along the way. I was still asking the question about what critical problem Go was solving. Perhaps it was obvious to everyone but me. The SREs had the answer, though. They had been writing all of their services in Python. I don't want this post to turn into a programming language war, so I won't go into the reasons in detail, but at Google, at least, services written in Python inevitably either a) turned into maintenance nightmares or b) ended up being rewritten in another language. Even knowing this, SREs kept writing services in Python - there were no other supported languages as simple as Python, and people who want to write integration layers and glue services together don't want to struggle with the complexity of a C++ or a Java.

Go, it turns out, provided a readymade solution for this. It was almost aggressively simple - they didn't have typesafe containers, all errors needed to be handled manually, they discouraged the kinds of heavyweight frameworks and libraries that Java developers used so liberally. It avoided the biggest pitfalls that came from using Python (again, fodder for another post). It came pre-loaded with everything you needed to run a production service at Google, and nothing that would distract from that. So, less than a year after the Go team succeeded in their goal of getting SRE to support their language on services in production, word came down from their leadership: services would no longer be written in Python. Their new language of choice was Go.

I finally got it. It took me a few years, but I finally got it. Go was there to save us from the nightmares we were having scaling Python. +2, LGTM, no notes, I'm Jeremy Manson and I approve this message.

When Sameer replied to my original post, I said I thought there was a lesson about the approval process to be learned about how Go evolved at Google. In short, it's this: our approval process exists to make sure that you know what you are getting into. People thought that the approval process existed just so we could say no - that we were making the bar seem impossibly high for reasons of our own. We weren't. We were telling folks what it really meant to get a language working in that environment. We were Marley's ghost, letting our Ruby or PHP or Scala proponents know that their best chance for redemption was to run the other way, as fast as possible.