Put the 'ease' back in 'Release'

The story is all too familiar. Your team is releasing their software tonight. It’s some new feature that everyone’s been waiting for. You feel good (as good as you ever do on the night of a release), and it's been a long time coming. This new feature has been through more than enough cycles of QA, and people on the team are ready to get this thing into production.

BUT - you’ve been here countless times before. That voice inside your head is still telling you “we won’t really know until it’s in production…”

You buckle up, dial into the release call, and brew a pot of coffee at 9pm because you know it may well be a long night. This isn't your first rodeo. You won’t learn if something’s really wrong until the software hits production and people start using it. It might not even be YOUR teams’ code or feature that has a problem, but we integrated a while back, so now we’re all in this together, one big integrated team sharing a common codebase, and one big release. Everyone's hoping for the best...

Sometimes things go smoothly. Sometimes they don't. If something’s not right, you’ve got to be on that call, where everyone’s trying to figure out whether we need to roll back, whether we should make a patch and roll forward, or whether we can live with this issue until the next release. Everyone's tired, there are more than a few people "not at their best" when the days are this long. Sometimes, 'reactions' win out over 'responses', and this is the time when we all know we have to exercise a little bit more patience than is usually required. Regardless, you’re probably going to need a lot more coffee, and you won't be going to bed for a while. Your "better half" watched their shows alone tonight, and they went to bed hours ago. But they understand; they know the drill. This is how it’s always been.

It’s “part of the job”.

You know what? It doesn't have to be that way. I promise you, there is another way.

Let's imagine a different world for a minute.

• Your team pushes a small change at 10 am, they're able to see that change in production in minutes. They try something, they're able to quickly see how customers respond, and they can revert it, or make a tweak or improvement minutes later.
• Your team delivers some tiny thing that's valuable to customers RIGHT NOW - long before the 'whole project' is complete.
• Instead of your boss asking you "when will this thing be done", your team is proactively demonstrating what value they've created for customers in the past week, how customers have responded, and how the needle has moved on that objective we've been targeting.
• Team members are able to leave work at predictable times, enjoy their evenings and weekends with their families, and being on release calls in the middle of the night is a distant memory.

It's a nice dream that's easy to get on board with, but to many people, it's still very much a dream. The path to getting there seems impossible to imagine. You KNOW it would require massive changes - in mindset, in tools, infrastructure, processes, maybe organizational boundaries. It seems like an easier proposition to just scrap everything and start over.

Well, let me tell you - you can do it. We did it. And frankly, you don't need a big elaborate plan. You just need two things: To make that dream a goal, and a shared commitment to continuous improvement.

If you're still doing middle-of-the-night deployments, and releases are ' calendar events' in your organization that people schedule their lives around, I'm sure the idea of painless, middle-of-the-day, multiple-times-a-day releases seems far out of reach. It was for us too, not that long ago. Releasing "on demand" was something we had wanted for a long time, and had put effort toward for years. We'd make progress, and then would seem to backstep. There was no shortage of desire, but consistent results always seemed elusive. We'd gotten to the point where we could release once a week, and we could squeeze out one more under stress if we needed to, but it felt like we'd reached a ceiling, and it was certainly not without pain.

In August of 2019, we gathered a group of people together who were all in some way responsible for the AA.com deployment and release processes for a day-long session to figure out how to release more frequently. In this session we came up with a goal that was literally laughable. The laughing told us that it was audacious, and yet it was definitely a goal that everyone found motivating:

"A developer on AA.com can see their code in production in less than one hour after pushing, with no human interaction required"

We mapped out the process and flows and tried to identify everything that had to happen to get code into production, we identified things that took a long time, or where there were long waits, or redundant sign-offs. We identified points of pain, and we also agreed that if we wanted things to be different, we had to be willing to challenge everything.

We started meeting every two weeks, and each cycle we'd have a single thing we wanted to improve, or a single pain point to remove. Nothing heroic, and only things we felt we could complete in the next two weeks. Early on, we added a caveat to our improvements. "...without adding more work". We knew we'd quickly run out of steam if we relied on heroism.

In less than two months, we no longer released in the middle of the night (that showed up EARLY as a pain point that everyone was motivated to remove). It made us a little uncomfortable at first, but we'd re-assessed the risks and the rollback strategy. We thought we could try a low risk release during work hours. Just one! We'd been able to make several tweaks to our release processes by then, and we'd already removed some of the pain. We'd removed quite a few human approval steps out of the process, for example our release manager didn't need to consult with teams on a decision to roll back. She could make that decision whenever she wanted, and pull the trigger. The conference call could come later. If we could just get through one or two releases this way, we'd learn a lot. And that's exactly what we did. We got through several releases that way, AND we learned. a lot.

This continued. We optimized the way we annotated change, by shifting that information into pull requests. We shortened communications, we shifted approvals, we removed things from the critical path that weren't exactly critical. Every two weeks, things got more lean. And every subsuquent release, we saw more and more small wins. Things got better and better. Continuous improvement became the norm.

One year later, in August of 2020, we realized we had actually achieved our laughable goal.

In the summer of 2020 we created the above video to share our success with other teams at American; We wanted to inspire them by telling our story about how we were able to go from big, painful deployments that happened once or twice every two weeks - to releasing code on demand, in the middle of the day, whenever code was ready, multiple times per day!

After talking this past year to others in IT and outside of American who have similar goals about release frequency, and having shared the above video with them, each of them has inevitably encouraged me to share the video with the rest of the world. I hope you are inspired by it and encouraged to continue to pursue dreams of painless software releases. What I love about the video and what I hope comes across, is the joy and freedom that comes across in these people's story. The journey is theirs, the achievements are theirs, and the joy is theirs.

• Is there a cool tech story there? Yes.
• Is there enormous business value? Yes.
• Does the organization have improved capability to adapt and respond? Yes.

But what I personally love about our story the most, is that people now enjoy their work more. Developers and teams can see their product making a difference immediately. People have more autonomy, they're able to have real impact with less friction, but most of all - they get to spend their evenings with their families and not on a conference call, jacked up on 9pm coffee with their fingers crossed :)

Some key points I'd like to make:

• We used the Improvement Kata to imagine and share a future state, and to make incremental progress with discipline. The Kata's empirical approach, along with the ability to experiment and learn along the way, and the discipline of only solving the most important 'next thing' is critical to successful change like this.
• This wasn't a "project". It was a shared goal for a group of people who each committed to make tiny improvements and getting together on a bi-weekly cadence. It wasn't anyone's "full time job". We found that it snowballed and built momentum along the way with each and every 'win' (and there are a lot of them with the improvement kata) and as a result, levels of engagement and forward momentum were high.
• Our "dream" / "definition of awesome" was achieved after almost exactly 1 year of the Kata. However, "releasing during the day" was a very early improvement that was achieved after only one or two months into the journey. It required no new tools, no new technology. You will be amazed at what you can achieve with much less investment than you've anticipated. Bring in only tools that solve your next problem, when the problem is directly in front of you, versus speculating about what you'll need ahead of time and overplanning.

If you're inspired by this, or if it sounds similar to something you've been working on in your org that's proving really difficult, I'd encourage you to use the Improvement Kata to start making incremental improvements. Have a dream that humans can connect with, and just find a way to make things better one small thing at a time. You'll be amazed at your progress if you can just commit to making things a tiny bit less painful each and every step of the way.

?...And get some sleep!