DevOps Rituals with Continuous Delivery

You’ve heard of the application economy: how software is eating the world, how Nike is not just a shoe company but is a digital powerhouse, and how banks think of themselves as a software company with a banking license. Uber has changed the way we use transportation. Research shows that companies that embrace digital transformation are 26 percent more profitable than their average industry competitors, receive a 12 percent higher market valuation, and generate 9 percent more revenue with their existing physical capacity.

Look at the folks at Tesla. A small but innovative car company with direct sales, no third-party dealers, and an all-electric motor, they found a defect in their car that caused it to roll off when the foot was taken off the gas. Tesla would have been faced with a massive recall, remember Toyota’s unintended acceleration recall. But Tesla put out a software update to fix the problem directly in the cars overnight. Consumers didn’t have to bring their cars in to the dealership and didn’t have to wait for an appointment and miss work; instead, a software update solved the problem. Imagine the speed, cost, and customer experience advantage Tesla has over Toyota, BMW, or any major car behemoth. This is a result of the application economy.

The need for digital transformation has led to the rise of DevOps. To enable self-service, software developers need to create the software that users will work with, and operations teams have to orchestrate the processes in the background to deliver the services and goods on the back end. Thus, we have the collision of the two worlds and the rise of DevOps.

DevOps is a set of philosophies focusing on various aspects of culture and process. The folks at Puppet Labs have put out a fantastic State of DevOps Report since 2014. With every year, each category — deployment frequency, lead time for changes, mean time to recovery, and change failure rate — has had a positive trend. This solidifies the stance DevOps plays on today’s software stage. Compared with their competition, high-performing organizations have demonstrated code deployments that are over 200 times more frequent, lead times that are 2,500 times faster, a mean time to recovery that is 24 times faster, and a change failure rate that is three times lower

If DevOps is a set of philosophies, then Continuous Delivery (CD) is the practical application of the teachings of DevOps. Embrace security, nonfunctional requirements, negative tests, performance, and disaster recovery from the get-go, and you’ll be adopting the tenets of CD into the daily regimen — whether that’s trunk-based development to achieve Continuous Integration (CI), or service virtualization to simulate applications, or testing high availability with Chaos Monkey.

Pillars of Continuous Delivery

People, process, tools: you’ve heard it all before. We have to strive to improve and integrate everything to excel. CD requires us to put these basic principles into practice. But we don’t have to do it all at once. Where you start and how you get there varies by organization. Sometimes it’s finding your biggest impediment and neutralizing it, or dividing tasks into small, manageable chunks of work. I always recall lessons learned on the golf course, from trying to master golf and failing miserably. It’s the little victories that count: with each hole, each drive, each putt, you continue to learn how to make the needed adjustments, and you get better.

Within CD, a number of methodologies and subdisciplines have arisen. All are designed to help transform your software development to impact your business and promote some fundamental paths. I’ve defined these in the diagram below.

These pillars become useful when trying to identify where an organization is currently focused and where it could possibly see the biggest gain with increased focus.

When I’ve asked businesses what’s important to them, a consensus of speed, quality, risk, and cost generally rises to the top. If speed is paramount, what is slowing down your software development life cycle (SDLC)? Is it a lack of CI due to unavailable 24/7 environments and working back ends, or to the fact that your deployment is a heroic manual effort hidden in the scripts that deploy software? If the risk of data breaches is going to cripple your organization, maybe your plan to not import production data is the key. Does quality take the cake? Then are you confident you have full test coverage of all functionality? Be certain and deterministic. Did you plan for negative testing? All this finds a place somewhere in CD.

A Practical Approach to Continuous Delivery

Well, all this is good, but you still may be asking yourself, ‘How do I take all these ideas and translate them into actual plans?’ This is what I see high-performing teams doing to take the principles of agility and put them into a practice that supports continuous delivery: 

• Develop on a cadence, deploy on demand – Employ all the agility principles, from two-week sprints to delivering minimum viable products, on-demand deployments, and production as needed.
• Shift everything left – Shift nonfunctional requirements (performance, integration, user experience) from late-cycle testing to the start of the SDLC.
• Create centers around behavior/data/model-driven development – For instance, visually represent the requirements in a model flow, or increase quality by reducing the ambiguity in requirement interpretation by various actors.
• Automate everything – From automated code reviews to builds, implement automation at every layer of the architecture stack, including automated environment configuration and, finally, automated deployments.
• Generate synthetic test data – Develop on-demand data for testing, when needed, and mitigate risk by avoiding reliance on production sources.
• Implement trunk-based development – Move from release branching to check-in to trunk frequently, in conjunction with using canary deployments and feature toggles.
• Leverage service virtualization – Increase efficiency and reduce artificial waits by applying high availability to test environments for development, and by testing via simulation of behavior, data, and performance of applications.
• Release frequently – Perform small-batch releases into production frequently in order to promote manageable releases, easier rollbacks, faster fixes, and frequent input from the end user.
• Solicit feedback – Do this by always monitoring and incorporating automated health checks.

The consequences of those practices lead to:

• Agile delivery
• Decreased mean time to resolution
• Increased speed to market
• Improved quality
• Excellent customer satisfaction
• Higher efficiency
• Reduced waste
• Reduced risk
• Low change failure rate

Attaining agility in the delivery of applications gives you the ability to be flexible and to pivot directly on feedback from end users. The overhead of making changes should have an insignificant impact on the overall effort. Instead of going through time-consuming approval and requirement development processes, feedback is taken directly from the users, prototyped, and released for feedback from the users.

A few years ago the folks at Twitter heard about Instagram and thought that integrating their two offerings might provide value to end users. They were not sure whether users would care about the integration or how they would use it. Using very little time and effort, they built a small subset of the integration and deployed it into production. By obtaining direct feedback from the user early in the dev cycle, they were able to validate the idea and use feedback to implement the features users wanted.

Mean Time to Resolution (MTR) also provides a way to measure the success of your efforts to become more agile. Continuous Delivery does not mean you have to continuously deploy to production every 15 seconds. But what if you need to do it faster than you’re doing now? You start by calculating the time it takes to change one line of code in dev and move into production through a normal cycle. Then you look at where the biggest slowdowns are and focus on addressing those.

Slow Down to Go Fast

Like many things in life, hard work and perseverance pay off. Why do I mention this? Because going on the journey of DevOps and CD requires not only a shift in the way we develop, how we test, and how often we deploy, but also a shift in culture and how we think about software. It takes some time to nurture all of these philosophies. You will slow down, but eventually the investment will come to fruition, and your company will be transformed into an innovative and agile organization that is able to compete in the new digital world.

I’m a huge Formula One fan. There used to be a time when going to the pit lane was the kiss of death. Strategies were developed to avoid or minimize pit stops. A few years ago, F1 reduced pit times to six or seven seconds, but that still caused a drag in the lap times overall. Today, three seconds is too long. Pit strategy is a system that tries to use to advantage on the track the frequency and duration of pit stops and how much gas (NASCAR) is used to fill the car. Drivers slow down to go fast.

To summarize the article from the pit strategy in F1: just as pit crews practice their art frequently to be perfect, practicing and frequently employing all CD strategies from the get-go makes your process more reliable.

The next wave is imminent: DevOps and CD. I think of it as DevTestOps. I heard the manufacturing line analogy from a very smart CTO I had the privilege of working with, who said the line should never stop. Think of the SDLC as the line, and the software as the car being built. CD is a way to keep that line moving. As I once heard, we must move from being the machine to engineering the machine.