2 out of “The 3 Ways” is not good enough people!

If you are not aware of “The 3 Ways”, then get hold of a copy of the DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois and John Willes. It’s extremely important that you do! If you are, then please read on, because our efforts to improve on only 2 out of the 3 ways is just not good enough.

The first and the third ways – The CI/CD pipeline

Much of our concentration has been on the first way, Flow, and the third way, Continuous Experimentation and Learning. Flow is about the capability to quickly deliver from an idea in development to a value producing system in Operations. Primarily implemented by Continuous Delivery using release orchestration, deployment and testing automation tools. The third way is primarily implemented using Continuous Integration via a rapid and agile cycle of develop, check in, build and test automation.

But what about the second way, Feedback? There is a certain amount of feedback as part of the third way, but this tends to be localised to paired programming and co-located agile squads, but that is not enough.

As part of our flow there are many other additional checks and tests that are required to be performed. Whatever the flow process you perform, either a series of testing stages or via blue-green deployments, there are ultimately two types of feedback that must be managed – Defects and Incidents.

The third way - Continuous Feedback

Defects are issues discovered prior to production, whereas incidents are issues discovered once a service is in production. Despite the respective impacts to business value being different, typically, defects being high occurrence/low impact and incidents low occurrence/high impact, the way they are processed is very similar – detect, analyse, advise, recommend and repair. This is our typical feedback flow.

There is a reason we must consider an approach to Continuous Feedback as importantly as we must consider Continuous Integration or Continuous Delivery, and it is this simple premise – issues happen!

When we consider Service Velocity, the speed at which we can enable, or recover, a service of value, it is not just the cadence of our flow, but also the cadence of our feedback. The longer we take to detect and resolve an issue, the longer it takes to provide or recover that service. In measurement terms, this means improving our Mean Time To Detect, Recover and Value (MTTD, MTTR, MTTV)

Decisions, decisions

All activities are defined by an input condition, an action, and an output response. When we look at how improvements are applied to flow, it is a case of providing as lean an execution as possible by reducing our flow to a few key activities and automation of those activities. In the CI/CD pipeline, this is made easier by always ensuring a single input condition (the previous activity completed as expected), a single action (pre-determined and automated within a tool) and a logical output response (pass or fail). In the instance of a pass, this is the input condition for our next flow activity. In the instance of a fail, this is the initiator for our feedback activities.

There are 3 key problems with improving feedback.

• Firstly, there is rarely a single input condition. For example, when a system fails, is it in a service that is due to be decommissioned or replaced? Is the value of restoring outweighed by the risk of change? Plus many other considerations.
• Secondly, the actions we are going to perform for detection and recovery are rarely understood beforehand with absolute clarity. Perhaps we are just required to restart a service, maybe perform a rollback or kick off our Major Incident Management processes. Perhaps even a combination of those.
• Thirdly, there will typically be more than two output responses. In its simplest form, for a defect, we may fix, workaround or do nothing, but invariably it will be combinations and degrees of those depending upon the input conditions and the actions we perform.

The key difference is that for flow, all actions have a decision gate that is logic based, which is perfectly suited for automation. The decision gates for feedback are risk based. We determine the actions we must take, to get to a response we want, based upon the input criteria, to provide us with the recovered service at the greatest value with the lowest amount of risk.

Orchestrating Feedback

This then brings us to the crux of the matter for feedback. The most important component of Continuous Feedback is also the component introducing the greatest toil, or waste, the human being. To enable faster Service Velocity through the orchestration of Continuous Feedback, we must integrate the human being into our feedback flow.

In my next article, I will explore more about how this is possible. I am also due to speak on this at the London Unicom Agile, DevOps and Testing event, re-scheduled to July 2020, and at the Stoke-on-Tech Meet-Up event, potentially in May 2020.

If, however, you are feeling a little impatient, or will not be attending those events, reach out to us at xmatters.com to understand how we can provide a platform that integrates the tools used to resolve issues, with the vital human beings that are required to implement a fix.

Stephen Walters, Solutions Architect, xMatters inc.