DORA Do's and Don'ts

Written by Matt Hawkes

In the spirit of beginning with the end in mind ... DORA is the hot-ticket framework you need to be aware of for measuring KPI(s) and success metrics for software development. If you read nothing else from this article, I encourage you to spend some time reading more about this critical framework.

"But, Matt,?you can find lots of "set it and forget it" tools that will collect these insights for you!" You might say, but just hear me out. Tools are expensive, and there is very unlikely a one-size-fits-all answer for your unique organization. If you are on a budget or working for a startup, you probably need to figure out how to collect them for yourself.

As a recovering JIRA Ninja (certified) and GitHub Organization Administrator, I have had the privilege of refining this framework at multiple stops so I am keen to share what I have learned!

Warning: what I’m about to delve into is not the “easy path”; however, it will help you understand your engineering organization's relative health. In this first article, I will explain what DORA metrics are and how to collect the first one of the four using GitHub.?

Let's go!

What are DORA Metrics??

DevOps Research and Assessment (DORA) metrics were created by Gene Kim and Jez Humble, along with Dr. Nicole Forsgren, when they published?Accelerate?in 2018.

These metrics were derived over the course of four years (2014-2017) by surveying the development community via Social Media, etc. They honed in on what are the most important metrics to measure software development quality in delivery using questions like the following:?

• What does it mean to deliver software, and can it be measured??

• How does software delivery impact organizations??
• In what ways does culture matter, and how do we measure it??
• What technical practices appear to be important??

Each year, using feedback from the previous year, Humble and Dr. Forsgren morphed the surveys to refine the data further and determine what was really important. They came up with four simple metrics that we can use to assess the health and performance of an engineering organization quickly:

• Deployment Frequency?

• Lead Time for Changes?
• MTTR (Mean Time to Resolve)?
• Change Failure Rate?

Using these metrics and a few tricks from GitHub and JIRA, we can quickly build these for any organization! Hooray!

In order to measure these effectively, we need to have some agreed-upon assumptions:?

• Pull Requests?are used in GitHub as the overall unit of work?
• JIRA?is set up to have a distinction in issue type between Bug, Incident, and User Story?

Collection Frequency?

I prefer to gather DORA metrics monthly, especially regarding MTTR, as it gives you a better data spread than per sprint. My goal as an engineering leader is to have zero incidents ever, but realistically, even the highest-performing teams average at least one incident a month. If you only have one incident in a month, and it takes a long time to resolve, that can really skew your metric.?

Metric #1 - Deployment Frequency?

Deployment Frequency is defined as how often changes are pushed to the production environment. This assumes you have a well-vetted?continuous delivery pipeline?that hopefully has feature flags in place to control when the code becomes active. Since most organizations using GitHub use pull requests, I like to calculate the number of merges in a given month using the?GitHub search query?(shown below for my favorite open-source project?Grails).

Now, if I run all of these in the search bar in GitHub, I get the following beautiful chart, which shows my deployment frequency since the start of 2023.?

By the Book?

According to Accelerate, the following ranges are established as a relative guideline for Deployment Frequency in 2016:?

• High Performers - One Demand (Multiple deploys per Day)?
• Medium Performers - Between Once a week and Once per Month?
• Low Performers - Between Once per Month and once every six Months?

My guideline for developers is to always push at least one PR to production per week. The best developers can push multiple PRs per day to production, but those are the real rock stars. The sweet spot is between 2-5 PR(s) per head per week. Using the Grails example above, the average PR(s) pushed monthly for the organization is 96.5, with around 24 PRs pushed weekly. What that means is if I have five engineers on this team, then I am ecstatic, but if I have 25, then not so much.?

There are, of course, things that can contribute to the overinflation of these numbers, but as a leader, it allows you to do some quick math to get an idea of how things are working without building or buying special tools.?

Part II - Lead Time for Changes?

The most complex DORA metric to collect, in my opinion, is the one I will discuss in the second part of this article. Stay tuned!

May the insights be with you!