Devops Transition: Process Retrospective
It's a giant monolith of code that doesn’t integrate with other applications. It has a quarterly release cycle that requires 3 bug releases to work. If it's SaaS-hosted there is a bonus prize of infighting between development, networking, compliance, operations. Support doesn't know to where to route tickets. People may lose their marbles on any given Tuesday.
Forrester published "more than 50% of enterprise businesses are now actively using internal or external DevOps services" which helps dissolve the above described scenario. However, adoption is limited to a few tools and teams because change is not easy. We can do better & I do advise product and engineering managers to push strongly for devops processes and tools. It'll lead to faster product development with less issues.
This is a story about how I’ve implemented devops cultures. It relies heavily on change management principles including Kotter's 8 Steps and other publications that I'll reference inline. Step 5 of this process is where the success or disaster sleeps. I hope this article helps write your own devops success story.
All views expressed are my own, and do not represent any employers or clients.
DevOps Implementation using Kotter's 8 Steps
1. Create a sense of urgency (1 month)
People debate whether a devops, or any change, initiative needs to come from the bottom-up or top-down. The answer is "both". When a problem is grossly unacceptable for all organizational layers then any person can be recruited into driving a solution. With devops transition, just rally around the problem of every product company - engineering can't release code as quickly as the business desires. In every conversation where the product team couldn't deliver a feature my response was "We need to change", " We can't grow fast unless we change", and/or "We could meet more expectations this year if we moved faster". The communication was soft yet clear. Persistent and without blame. Exasperation without tension.
How does this play out?
The Top-Down Urgency / Request to Change
My boss wanted a product change in an impossible time. While I couldn't guarantee the requested delivery date if we changed, I did guarantee that the product organization wouldn't deliver on time without change.
Bottom-Up Urgency / Request to Change
The SaaS engineering team was exhausted from technical debt, network changes, 2am outages, repetitive manual testing tasks, and the overall blame-game that happens when a development team is blind from, or semi-responsible for, production. I held meetings with the most influential members of engineering and operations to describe the current state in the most dreadful manner. I asked, "What can we do to be better?" Everyone spoke, and a few people mentioned devops principles.
2. Build a guiding coalition (1 week)
The next steps required a new message with a select few friendly people - 6 engineers and my boss. "We are going to improve development speed and be happy. You in?". I didn't explain yet what the vision was, or any details. I just wanted to find a small group of people who believed we could do better and were willing to change.
3. Create a Strategic Vision (2 weeks)
While step (2) was occurring, I was also drafting the DevOps Vision. My aim was to give the following PowerPoint slides countless hours of airtime to get buy-in from stakeholders.
Slide 1 - mostly a copy/paste of concepts from the book "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" - an excellent book.
Slide 2 - textbook Agile and Lean principles
I had an idea of what people, processes, and technologies would be needed (each bullet on the devops slide can be 1-3 unique technologies and a few processes) but I didn't go into detail at this stage. I first wanted to test the vision slides with the coalition of change-enablers.
4. Enlist a volunteer army (2 weeks)
There is a 3-part strategy to buy-in from a group of people:
a. First, pre-wire individuals
I spoke with each person individually to explain the vision slides and listen to feedback. Sometimes my conversation was an email or chat thread, and other times a 1:1 meeting. I wanted the private conversation to ensure the person understood & supported the content. After a conversation the platinum question would arise, "Do you think we can change?" I revised the slides 100 times to until each person said "Yes, we can do that".
b. Then, connect the individuals to consensus
After the individuals were each aligned I made a larger meeting with the 7-person coalition. The meeting agenda was "You have all seen these before… and a think we are all aligned… but I want to walk through the vision slides again to see if there's any more thoughts… what do you guys think?" Success is when everyone nods their head and voices support. Failure is if someone displays resistance to change; it means I either didn't pre-wire effectively or should have removed them from the coalition. I was lucky enough to have a consensus that devops what was the solution.
c. Lastly, make the group's vision public
After the coalition had my back, I started exposing devops principles to as many people as possible: up, down, and sideways on the org chart. Anyone within ear shot heard my pitch about the pains of the product and how we are going to improve. I wanted the water cooler conversations to start, and the influential coalition of engineering leaders to start advocating for devops in those private discussions. A call for the devops implementation became inevitable.
5. Enable action & remove barriers (2 - 4 months)
This stage is the most critical because strategic implementations are chewed apart by existing cultures. Implementation is where people fail. There may be some quick wins, but "mission accomplished" is a long war often lost because of a few dissenting individuals.
Recommended Leadership Actions
a. Reset the Culture
I did a reorganization of the engineering team to ready the culture for change. Nothing too big, just shuffled developers to the engineering area they had previously expressed interest in. It was enough to reset the existing team norms so that they would have to establish new ones, but not so much change that all intellectual capital was lost.
b. Go Live. Go Big.
We reset all teams' sprint names to "DevOps Sprint 1". The sprint goal was to implement the 10 DevOps principles shown in step 3. Many processes were set in the beginning stating "this is a version 1 process. Try it this way and then in the sprint retrospective we will iterate to make it better." This was a little bit of forced adoption so that the use cases and supporting tools would be more tangible. The teams could choose their own technologies to support the use cases. There were no new hires, but many new responsibilities of deploying and troubleshooting the production environments. I was expecting that we didn't need new people because the total FTE count could go down with efficient devops - and this turned out to be true over the next several months.
Outcomes from Empowering Employees
Two cultural changes immediately occurred in the teams when we finally said "we are devops":
a. Increased Employee Engagement
Voluntary turnover in engineering dropped to 4% YOY during the transition. Everyone was challenged and engaged with a vision that would make their quality of life better. There was collaboration, POCs, and mistakes. Slack channels filled with new ideas and interesting blog articles. Team building occurred faster than a weekend of trust falls or Thursday office binge drinking; this was real work and the outcomes were noticeable.
b. Enter the Valley of Despair
The Valley of Despair model explains the negativity that occurs shortly after a change is implemented. It is also the step where most change initiatives fail. The team gets frustrated, leaders start to question their decisions, business velocity make take a noticeable dive, then leadership loses their nerve. When this happens, the changes may be backed out and everyone curses management.
The quality of leadership defines how long the team will be in despair, if the team crosses the valley, or if they roll back the change.
Valley of Despair + Kotter's 8 Steps
Push through despair with direct communication and high expectations:
- When the team falters, push them harder & reiterate your commitment
- When a manager falters, push that individual harder & reiterate your commitment
- Remove any people who decide to become an obstacle. In my case, 5% of the team became an obstacle to change and were removed from the organization. There are several articles published on the skill gap for QA to adopt devops, so I won't repeat it here.
- Don't ignore despair. Embrace it to overcome.
After a month the teams were firmly driving into the valley of despair. The selection & deployment of devops tools took more time than people hoped. New rules required bug fixes be deployed with a CI/CD pipeline & that took extra time "We can fix this bug in 3 hours if we just did it the old way instead it took two days to set up all the CI/CD infrastructure for a given project". Some people felt worthless because they didn’t have to skills to configure a tool via command line, which put additional pressure on the tech savvy resources. Several months later one tester wrote me:
"I wanted to quit a couple of times. I got frustrated…even though everyone kept giving me advice and being patient, which really helped me out..."
The Valley of Despair is depressing, and the way to success is firm guidance and celebration of wins while being depressed. Here's how two different teams succeeded:
A story of two teams, and how to manage/mismanage despair:
"Team 1" (2-month valley of despair)
Engineering members started to communicate their doubts about devops. Another stakeholder of the development team (an operations director) and me had hard conversations with the engineering team early and often. We reiterated our expectations, the goal, and helped the team think through challenges. As leaders, we made sure to not show any signs of quitting the devops change journey. Every time we saw a team member, or the team, exasperated we quickly had a private conversation to bring them back into the journey. Consequently, after 2 months of despair the team suddenly came together. The sprint review where everything clicked was noticeable because everyone was smiling. The velocity increased 4x from the previous sprint. We knew we were going to somewhere great, and no one wanted to go back (more about this on step 7).
"Team 2" (4-month valley of despair)
As a leader I didn’t do as well managing the change journey with this team and the result was noticeable. For one, I was the only leader advocating the devops change with this team. I was the only voice against a cackle of frustrated engineers and my voice wasn't as persuasive compared to the dissenters. Two, this team had to design a complicated front-end test automation strategy. I had a lot of empathy for the challenge, and therefore didn't push as hard when people became frustrated; this was a mistake because it let resentment and pushback simmer longer than needed. My empathy became a weakness that rallied dissent on devops principles labelled as "too hard". It took 2 months for me to realize my mistake and to finally start publicly & frequently reiterating a devops commitment. The team came together two months after I changed my own behavior.
I tell this story about Team 2 because this could have easily been a devops failure with a roll back of tools and processes. Perseverance and learning paid off. Stay strong on the devops journey.
6. Generate short term wins (2 months)
Each sprint had short term wins that were celebrated. People were encouraged to email the entire team when a new component was completed, or a service went to production. We celebrated as we dove into the valley of despair, grudgingly celebrated more wins as we wallowed at the bottom, then even more wins as we rose to the top. Sometimes we even made up wins just to feel better.
7. Sustain Acceleration 1 month)
Culture changes happen like a canary build. Someone deploys a change and the positive effect migrates around the environment. Smaller wins roll up and out into big changes. For us the outcomes were decreased volume of support tickets, increase in environment uptime, adoption of microservices, & production releases occurring multiple time per day with no fuss. More time became available for further improvements because there were no fires to put out. The product roadmap suddenly looked achievable.
Once the team landed in this step I backed off being the change driver, and instead became the cheerleader "Did you hear what Jon Doe did? He figured it out! Ask him about it because it may help you do X…". I was no longer talking about what needs to be done, instead I was telling people what had been accomplished.
The sprint retrospectives did their work and improvements started to occur on their own. People were discussing technologies and processes I had never heard of, nor could I keep up with. The team was now sailing on their own.
8. Institute Change (Probably forever)
News spread around the company about our success. Other teams and leaders became interested in having their own journey, or sharing lessons learned, to a better operating model. Engineering members within my team became informal advocates and subject matter experts for others. Concepts were firmly anchored in this stage. Engineering was firmly on a new cultural ground.
Takeaway
DevOps principles are growing. In 2010 I remember having a simple Bamboo CI pipeline, and now there are several helpful processes and tools available to speed up development and operations. Use them. As a product manager and former engineer, I love to see customer value delivered easily. If you’re thinking about the shift to devops, I hope this article helped. Feel free to reach out to ask questions or to bounce an idea.
Good luck.