Metrics That Drive Performance for CTOs and CEOs

Have you ever felt like driving a car without a dashboard? That’s how it feels to lead a technology organization without the right metrics. Let’s dive into the metrics that can guide your journey.

DORA metrics are great speedometers and warning lights for your technology organization. They tell you your current status and how things are trending and operate at a level that non-technical teammates can easily understand. Those metrics are great for reporting to the board and your peers.

But they don’t tell you where to look under the hood to find opportunities for critical improvements. For that, you need a different set of finer-grained metrics, leading as opposed to lagging metrics, and metrics designed for you and your leadership team to make changes, optimizations, and decisions.

If you thought your teams were underperforming, where would you look to make improvements? How would you know if your changes were having the desired result?

How to Lay the Groundwork for Effective Metrics

As you go beyond DORA with your metrics program, you will likely find more resistance from your team. At its core, software engineering is more craft than engineering and brings into play many variables and elements out of your, and your team’s, control. Meaningful measurement is challenging, and if you don’t proceed carefully, your team will see all of the risks and recognize little to none of the benefits. So, a few points before we get to the details.

Know your why

You should know why you want to measure your engineering team. You need to consider your audience, their goals behind the ask (if they asked), or your goals in providing the metrics. Ideally, the audience for these metrics is yourself and your leadership team. The metrics I discuss below are ideal for anyone responsible for your product delivery function and the efficiency and effectiveness of that function. These are actionable metrics but also metrics that require a detailed understanding of the process to evaluate their meaning and how to react. You are likely to want other metrics for other audiences and goals.

Process not people

Your focus should be on measuring the process rather than the people. The goal isn’t to discover who your most frequent committer is or who is the slowest coder. The goal is to find where the process is breaking down or the team is bogged down by following the process and make process changes. Metrics aren’t for leaderboards, they aren’t for comparing people, and they certainly aren’t for making people changes. Metrics are for tracking progress over time and making process changes.

Teams Before Individuals

Software development is a team sport, and your approach to metrics should reflect that. Measuring the process at the team level is more likely to give the insights you need to optimize the team and the process. Individual metrics can help inform one on ones, but for optimization purposes, it is best to focus on optimizing the team, not the individuals.

Data Informed, Not Data Driven

It’s still true that the goal is to be data-informed rather than data-driven. As with DORA metrics, the numbers are the start of a conversation and not the end of the process. Especially for leading indicators at the team level, the numbers are great for sprint reviews and driving conversations about where to make process changes.

Trust

These metrics can get contentious. Your team may react poorly to this level of detail being tracked. The introduction and rollout of these metrics benefit significantly from high trust between your leaders and your teams. If there is a history of distrust or abuse of data to the detriment of the teams and individuals, it may be best to move very slowly and demonstrate that your leaders will use the data responsibly before the widespread rollout of a full suite of metrics. Start with one metric, clearly aimed at the process, not the people, and tracked at the team level.

Are you and your leadership team aligned on the goals of your metrics program? Are you aligned on the focus and how you’ll make changes informed by the metrics? How will your teams react?

How to Build Empowered Autonomous Teams With Metrics

I’ve found a handful of metrics particularly useful in measuring how a team’s software development process and practices support or get in the way of fast, frequent deployments. These metrics are great at pointing your team in the right direction to make tweaks and adjustments for improving team performance.

These metrics are empowering for the teams. The teams build mental models and linkages between what they do daily and the results we see, both in production for our users and in the DORA metrics. Without that linkage, it’s easy for teams to think they have no control or impact on the DORA metrics. These metrics make it very clear that the DORA numbers directly result from what the team is doing daily.

The other critical mind shift I’ve seen in teams is a shift from focusing on their own work first to focusing on helping the team get work over the finish line. This isn’t just a shift from “me first” to “team first”. The critical shift here is from starting to finishing. This shift is critical for agility, speed, throughput, and quality. Using these metrics, the team concludes correctly that “work in progress” is the enemy and that time, energy, and focus spent on getting any work in progress, theirs or others’, over the finish line is the best use of anyone’s time.

Where is your team’s focus? Are they inclined to focus on their own work first, or are they team-oriented, focused on adding customer value?

Our teams are self-sustaining, with all the skills and resources they need to execute the entire product development lifecycle. They know their objectives and are given broad leeway to deliver them however they see fit. Part of that empowerment is determining their own development process (with a few guardrails) and the freedom to tweak, customize, and optimize that process to fit their needs best. These metrics guide that optimization process, pointing them to problem areas and providing feedback on how successful those efforts are.

Are your teams empowered and encouraged to “own” their processes and tools? Are they given the freedom to experiment?

Aging Report

The first tool we implemented that helped teams move the needle was an aging report for active stories. We used a tool that gave our teams insight into the current JIRA status of every story that was either committed to in the sprint (for our scrum teams) or that was in process (for our kanban teams), how long it was in that status, and a projection of how long until it completes based on that. Teams would use this report daily in their standup meetings, and it was very easy to see when a story was stuck. It allowed the teams to get those stories unstuck quickly and before the impact worsened. The report drove conversations about extra help or attention stories might need in later stages…particularly demanding QA needs, for instance, or if coordination with another team would be required. The teams quickly came to expect that stories would move through the system quickly, and any stories that didn’t, or looked like they might not, got discussed, planned for, and addressed.

Code Reviews

Code reviews are one of the most common areas where teams lose cycle time, slowing down change lead time and impacting the number of releases. Two metrics our teams used to keep an eye on this part of the process were “pull request size/complexity” and “time to first review.”

Many tools in this space have proprietary metrics for measuring a pull request’s size and or complexity. These metrics allow you to compare the relative mental capacity, time, and effort investment needed to review a change. “Lines of code” is a component, but the complexity of the change and how far-reaching the impact is plays a more significant role. Some complexity can’t be avoided, but teams quickly learn that frequent, small PRs move through the system faster and more reliably than large, complex, and infrequent PRs.

“Time to first open” tells you how responsive your team is to pull requests. As in lean manufacturing, wait and queue times are wasteful and costly: time is spent while no value is added. Teams can quickly understand that shortening the time to first review is a very easy way to gain cycle time, reduce the likelihood of errors, and increase deployment frequency. Most teams start this improvement process by reviewing the aging report from above during daily meetings to identify code reviews that need to be completed. As they gain traction, faster notifications are necessary, usually implemented with Slack notifications. WIP limits on the “in code review” status can be a great tool in getting the team focused and prioritized on getting the team’s work to “complete” rather than simply advancing their own work to the next step.

QA

The QA process is often revealed to be another source of delays in cycle time and presents opportunities for improvement. Tracking test coverage, test run time, and the balance between automated and manual QA efforts can help diagnose how to improve this critical area. QA approaches vary wildly from environment to environment and often with good reason. Despite the variations, improvements can typically be addressed through 1) more automation in the CI/CD pipeline, 2) feature flags that allow separating deployment from release, and 3) WIP limits for manual testers. As with the code review, low WIP limits for the QA process will focus the team on completing rather than starting work.

Balance

Metrics tracked at an individual level can considerably impact understanding of team dynamics and performance. Specifically, metrics around code reviews, commits, and deployments can identify cases where a team is over-reliant on one or a few key players. This typically arises in teams with a wide range of skill levels when a few experienced engineers are implicitly or explicitly expected to review all the code. It is also common in teams responsible for legacy systems, where one or a small handful of people know a particular subsystem well, and all work in that area falls to them. When looking at the data, the impact of these sometimes subtle bottlenecks becomes apparent, and the value of investing to remove them is equally obvious.

I’d love to hear stories from other CTOs and CEOs…what metrics are your teams using to optimize performance? What metrics have you found that best address your team’s specific opportunities for improvement?

What was your experience with rolling out metrics at the team or individual level? What lessons did you learn through the process?