#InsaneDevOps Ep2: Max Out Cardio (DevOps Performance Benchmarking) #ASODR

This is the first in the 'Insane' phase of the Insanity Max series and it’s a full body workout. Under the instruction of Shaun T, the goal is to max out - i.e. work as hard as you can until you can't… then take a moment and carry on. The work really begins after you have maxed out.

One of the purposes here is to identify your performance benchmark - where do you max out and what exercises cause you to max out? You have just worked your heart, lungs, upper and lower body - you will know what hurts, where you slow down and what gets really sore after this.

You will return to this many times and if you are disciplined, you will see dramatic improvement.

DevOpsGroup Discovery Process

At DevOpsGroup, this is the equivalent to our Discovery Process. We have partnered up with DevOps Research Agency lead by Nicole Forsgren, Jez Humble and Gene Kim and we ascertain from the teams we are working with how they perform against the 21 capabilities identified to predicate Business and IT Performance. Following research across over 30,000 responses from over 2,000 unique organisations of all sizes from all industries (as depicted in the 2018 Accelerate State of DevOps Report), a team can find out where they compare against industry averages and against those classed as 'high performing' - Also, we identify which of the capabilities are constraints and are holding you back.

This survey is conducted across the organisation that is collectively responsible for business outcomes or software delivery - this will include Development, Test, Operations, Security, Infrastructure, Database, Analysts, Architects, Change Management and Release Management teams. It is important to obtain a holistic view and there needs to be enough people surveyed to make the results statistically viable. This will help identify where the organisation lies in relation to the industry, as defined by DORA:

This is our Max Out Cardio for your organisation - lets put you to the test and see where you max out at and then, using a toolkit of tried and tested workouts (we call them ingredients), we then look to address these constraints.

Max Out Cardio will test 4 areas; stamina, strength, agility and power. DevOpsGroup and DORA will look at four capability areas; Technical, Process, Measurement and Cultural.

Unlike Shaun's fitness programme which measures through your max-out time and key measurements such as waist size, bicep circumference and weight, Forsgren et al survey the organisation based on Likert-type questioning where participants either rate their answers between strongly agree to strongly disagree. The questions are specifically worded to stimulate a focussed response. An example of a question would be "We are primarily notified of failures by reports from customer" - the respondent will then rate how strongly they agree or disagree with this statement. The survey responses are then collected and the outcomes show how the organisation performance against the industry and high performers. the output also provides the relative position of each capability against each other - hence showing you the bottlenecks.

Following the survey, DevOpsGroup will conduct a series of workshop aimed to review the following; Goals, Work and Flow

Goals Modelling Workshop

The purpose of the Goals Modelling Workshop is to establish a common understanding of the business and department objectives and determine if these are common and aligned across different teams, roles and levels of the organisation.

During this session, we seek to establish a picture of where each team feels they want to be and what they are trying to achieve.

By mapping deliverables to a business goal, it becomes easier to align the business and IT departments. This simplifies the prioritisation process and enables the ability to limit work-in-progress, which are critical to successful IT delivery.

Work Identification Workshop

The purpose of the Work Identification Workshops is to establish what work is done, as well as how much work the organisation and teams are managing. This is also an opportunity to take a deep dive into the constraints highlighted by the DORA survey and layer in the necessary context to fully understand them. 

Flow Workshop

The purpose of this workshop is to establish how work flows through the organisation and teams. It uses a technique called Event Storming, which enables visualisation of the workflow for a specific use case in the system. Once the flow of events has been modelled, we facilitate an open-discussion on how the flow of work can be optimised through the system.

Event Storming is delivered in a workshop format to analyse complex business domains quickly. It is a useful way to do rapid "outside-in" domain modelling, starting with the events that occur in the domain instead of a static data model. Run as a facilitated workshop, it can identify key domain events and place them on a timeline. This identifies their triggers and explores their relationships.

DevOps Capabilities Analysis

Technical: This includes practices that are important components of the continuous delivery paradigm, such as: the use of version control, test automation, deployment automation, trunk-based development, and shifting left on security.

Process: The software industry has borrowed many ideas from lean manufacturing which improve the way we work. Examples include: decomposition of work (allowing for single piece flow) and streamlining change approval processes.

Measurement: These include the use of metrics to make business decisions and monitoring tools.

Cultural: Our model includes measures of culture that are indicative of high trust and information flow, the value of learning, and job satisfaction. These measures have all been shown to be predictors of IT performance and organisational performance. Some of these measures are also leading indicators for technical feature success; that is, when people start fighting, technology starts breaking.

Key Outcomes Assessment

Completion of the surveys, together with the workshops should therefore result in the performance of the organisation against key performance metrics. Based on practices such as Lean, Accelerate identifies the follow as the key performance metrics in the delivery of software.

• Lead time: For the primary application or service you work on, how long it takes to go from code commit to code successfully running in production or in a releasable state.
• Deploy frequency: For the primary application or service you work on, how often code is deployed.
• Mean time to restore (MTTR): For the primary application or service you work on, how long it generally takes to restore service when a service incident occurs (e.g., unplanned outage, service impairment).
• Change fail percentage: For the primary application or service you work on, the percentage of changes that result in degraded service or subsequently require remediation (e.g., lead to service impairment, service outage, require a hot x, rollback, x forward, patch).
• Burnout is physical, mental, or emotional exhaustion caused by overwork and stress. It can make the things we once loved about our work and life seem insignificant.
• Deployment pain is the dread and disruption felt when pushing code to production. Our research shows us that more painful code deployments are correlated with poorer IT performance.

Visualisation of Outputs

The team is then plotted on charts that show their relative position against organisations in their industry and against high performers. This gives an understanding of relative awareness but this on its own is of little value.

Each of the above capabilities are also plotted on a chart where the two axis are; capability strength and impact on software delivery performance. All the capabilities are then plotted relative to each other - so there always a capability in the extremes of each axis - what this then does is visually identify the capabilities that, relatively speaking, are of low strength and have a high impact on delivery performance.

The challenge is therefore, to address the low strength, high impact areas first. The recommendations that result from this study, borne out of the research and statistics observed by DORA and the experience of DevOpsGroup, focus on the areas that will make the biggest difference.

A team will then execute corrective actions on these constraints and in a few months time, repeat the exercise to see how the constraints have moved. Just like in Max Out Cardio, with each week that goes by, you will see which area of your physical capabilities needs more work.

In theory, you could keep repeating these every few months, constantly addressing the constraints and thereby creating a learning organisation with improvement at its heart.

Misguided Performers

Warning: there are many people who do the workouts who look like they are doing well as they are not maxing out until late on in the workouts - however, when put to the test they cannot even do 5 mins under pressure.

The 2018 Accelerate State of DevOps Report have found that there are organisations that suffer from a cautious approach:

"Our cluster analysis this year shows a performance profile that matches this misguided approach: relatively low speed (deployment frequency and lead time for changes) and a better change fail rate than low performers. However, this cluster also reports the longest time to restore service.

Developing software in increasingly complex systems is difficult and failure is inevitable. Making large-batch and infrequent changes introduces risk to the deployment process. When failures occur, it can be difficult to understand what caused the problem and then restore service. Worse, deployments can cause cascading failures throughout the system. Those failures take a remarkably long time to fully recover from. While many organisations insist this common failure scenario won’t happen to them, when we look at the data, we see five percent of teams doing exactly this—and suffering the consequences.

At first glance, taking one to six months to recover from a system failure seems preposterous. But consider scenarios where a system outage causes cascading failures and data corruption, or when multiple unknown systems are compromised by intruders. Several months suddenly seems like a plausible timeline for full recovery. 

Customers may be able to use the system within hours or days, but engineers and operations professionals are likely investigating all the contributing factors for the incident, checking for and remediating data loss or inconsistencies, and restoring the system to a fully operational state. (The crash of healthcare. gov in October 2013 is a particularly high-profile example of this scenario.) When systems are compromised by intruders, investigation and remediation can take even longer. These periods of incident resolution are intensive and exhausting. When they happen multiple times a year, they can quickly take over the work of the team so that unplanned work becomes the norm, leading to burnout, an important consideration for teams and leaders." Accelerate State of DevOps, 2018

Pressing Play

So, you have done your test, you are sweating buckets and can taste blood in your mouth - your heart is pounding, you are exhausted and have no idea how you will complete the series… you are not proud of your measurements as these are not aligned to a 'healthy' person of your gender and age, and you know if you do nothing, your health is at risk and you will not be the best you are able to be.

Take your measurements, develop your motivation and start focussing on the areas that will make the biggest difference.

Shaun T says that pressing play is the start to a healthier lifestyle - if all you can do is one press up today, then that's ok - do 2 next time!

If you need help, Contact DevOpsGroup to conduct a DevOps Discovery underpinned by the DORA research and assessments and let us help you understand where you need to improve - [email protected]

More in this series:

• #Insanity Ep1: Insanity Max 30 (DevOps: Stability AND Throughput)
• #Insanity Ep2: Max Out Cardio (DevOps Performance Benchmarking)
• #Insanity Ep3: Max Out Power (DevOps Processes and Measurement)
• #Insanity Ep4: Max Out Sweat (DevOps Technical Capabilities)
• #Insanity Ep5: Max Out Strength (DevOps Culture)
• #Insanity Ep6: Friday Night Fight (Scale)