A practical guide for measuring your R&D

Introduction

Measurement has become a standard requirement in management. Most companies today have a set of metrics for different focus areas such as sales growth rate, customer satisfaction, employee satisfaction, ability to execute and many more.

For R&D, metrics can help track quality, efficiency and execution so it’s really important to implement them.

Designing metrics can be challenging, balancing usefulness with cost-effectiveness. This sometimes leads R&D managers to avoid them altogether.

In this article I am covering design principles for creating useful metrics for R&D including some real-life and concrete examples. After reading this article every R&D manager should be able to implement a small set of metrics that are both useful and easy to adopt.

Characteristics of Good Metrics

Start with Goals

Like many other things in life it’s a good idea to define goals before jumping to metrics. In general, the main goal of metrics is to be able to identify trends and lock down on areas for improvement. This is very general however. You need to be more precise in defining what it is that you want to excel in. For example, you might want to have a measurement of your product quality or the efficiency of the development process. Some goals can be derived from the company strategy. For example, if your company wins market share by duplicating capabilities between verticals with slight adjustments, then R&D should excel in doing that so this can be a good choice for a? goal.?

After you decide on a goal, come up with KPIs for this goal that can indicate if you do well achieving it. For example, if your goal is to have an efficient development process, it would make sense to require a short average build time. You should define “short” numerically to keep everyone in agreement on whether a measured KPI is in the good range. You can also define multiple levels such as “Bad = >15 min”, “Acceptable = 5min - 15min”, “Excellent < 5min”. For some metrics you might not know what number is good but you can get a sense for it after experimenting for some time by comparing performance of different teams, examining the variance, etc. It’s also recommended to use benchmarks when they are available.

Start with no more than three goals in the beginning and design metrics that you think help you measure accordingly. Each of the goals can possibly be covered by multiple metrics. Examine over time whether the metrics provide reliable description of how well you achieve your goal. Consult with the development teams and get their feedback. Adjust, until you're satisfied and only then add more metrics. Check out the “Real Life Examples” section to get end-to-end examples of how it works.

Metrics Drive Behavior

Metrics are essentially success criteria so by applying them you define what success looks like. Your employees don’t want to fail so they will try to be or appear successful given the success criteria. While metrics can incentivize desired behavior, they can also lead to unintended consequences. To demonstrate that I’d like to give you an example from personal experience. At one of the companies I worked for, developers were measured for the number of bugs assigned to them. At the same time, testers were measured for the number of times a bug they opened was eventually classified as “not a bug”. The idea behind these measurements is clear and at first impression seems sensible. In order to speed up development, developers should do their best to not throw code “over the fence” without testing. Similarly, testers should make sure that they only report real bugs. The impact of these metrics on R&D however was counter productive. Every new reported bug ignited negotiation: is it really a bug? Maybe we can classify this as an enhancement? Maybe it’s someone else’s bug? And so on… On one side, the developer was reluctant to admit of a bug and the tester was reluctant to admit it was inaccurately reported. You can imagine how much time was lost and the amount of frustration on both ends due to these metrics that aimed at increasing efficiency.

The lesson from this story is that you need to think about the behavior that a metric can lead to and try to avoid unintended consequences. Hopefully you hire people that want to be successful and so my recommendation is that you use metrics that help them steer towards success. Metrics are best suited for measuring progress, not individual evaluation

Objective Metrics are Better

It’s better to have metrics that do not depend on point of view or on any arbitrary decision. This is the reason that Story Points are not considered a good metric for productivity. Story points are used for planning purposes. If you start using them as a productivity metric you will most likely experience a Story Point Inflation and your productivity meter will rise high with no correlation to productivity increase.

Real Life Examples

In this section I will cover several common goals and suggest good metrics for them. As mentioned above, we start with goals: we want to achieve a high quality product, an efficient development process and productive teams. How can we measure these??

Product Quality

Measuring product quality is a very common requirement and there are many possible metrics. For brevity, I’ll discuss three here.

1. Rate of customer support tickets that required patch. Count the number of incidents that customers opened that ended up getting fixed. These are undoubtedly quality issues and because they come from customers they are objective in the sense that it’s out of your control. This information should be collected periodically. The data collection frequency depends on the product deployment characteristics (e.g. SaaS, on-premise, hosted). For on-premise products you may want to collect this information per release so you can see quality trends by releases. While on-premise products may allow measuring this metric once a month or even less often, in SaaS products you might want to measure this more often.? You may want to normalize the number of cases by the total number of active customers so that the metric does not falsely reflect quality reduction due increase in number of customers. You need to define what is an acceptable number and at what point you’d start taking action due to “Poor” quality. Of course you can decide to take action if the trend shows that you are moving from “Acceptable” to “Poor” consistently or rapidly.
2. Percent of production builds reverted or urgently replaced. This metric can be relevant for a product deployed on customer premises as well as for SaaS. It refers to cases when you had to act quickly in order to prevent customers from experiencing a severe bug and you did this by replacing a build that already passed all your quality gates and went to customers. This measures high severity bugs that were discovered shortly after releasing and it indicates short time between failures with high probability which is not a good sign. So in this metric you’d want to score very low.
3. Amount of “aging” pending bugs. Many organizations keep all unresolved bugs in open status hoping for better days when they will become available to resolve them. Allow me to borrow a quote by Robert C. Martin: “the goal of Agile is to kill hope”. Robert C. Martin referred in his quote to planning and how burn down graphs tell you early on that some stuff will slip out of the sprint and so you need to adjust as soon as you discover it. It is the same principle with bugs. Some bugs will never be in priority for fixing and you better put them in the archive and stop juggling them. Then, you remain with a list of bugs that you or some other stakeholders are unwilling to archive (close). They have to be fixed but maybe not right away… How can you keep yourself honest that you do not keep delaying these bugs forever or too long? How do you know if the number of such bugs increases over time? If it does, this means your development process creates more must-fix bugs than it resolves and this could be a reasonably reliable indication for quality degradation. In order to measure this metric you need to define a standard maximal time to fix. Let’s say every bug should either be moved to the archive or fixed within 2 weeks from the moment it's created. All other bugs are considered “aging” and you can count them on a weekly basis and review them periodically with your management forum (managers and team leads in R&D).

Development Process Efficiency

There are many potential indicators of development process efficiency and you should focus on areas that benefit your organization. For example, if you plan to increase headcount, you can measure the time it takes to get a new employee up to speed and try to reduce it. If you have pains in testing or in building time you can focus on measuring in these areas. Here are some examples of metrics that you can use or get inspiration from them.?

1. CI/CD pipeline execution time. You can measure average times and variance of commonly used or important CI/CD jobs. By measuring, you can identify trends and even be alerted in case of sudden degradation. Measuring the duration of sub-steps in the pipeline allows you to focus on the longest steps which are the best candidates for improvement.
2. Developer procedures duration. This metric refers to the efficiency of an individual developer working in his/her personal development environment. In different technology stacks developers go through different procedures when developing code. For example, some programming languages require compilation while others require only re-packaging or restarting of a service. You should strive for development procedures in your organization to work within the range of known benchmarks so check for potential areas for improvement and measure them. For example, if a Java or C++ developer makes a change, what is a good enough time for them to be able to rebuild and test? You should document the actual time and compare it with the desired time. Track progress to monitor improvement effectiveness.
3. Rate of build failures. From time to time developers merge code to the public branch e.g. master) and usually there is a pipeline that automatically builds and tests the integrated code. When such a build fails it slows down the development process. So it is worth examining the failure-to-success ratio. An increase in this metric is a warning sign that you need to adjust your processes

Development Team Productivity

This is the holy grail of metrics but because it is tricky, managers that choose to start with implementing productivity metrics sometimes give up metrics altogether. The problem with measuring productivity is lack of a valid reference to compare to. Whatever your team is working on is unique and not comparable with any other work done by any other team ever. So, what can you measure and what can you compare your measurement to if it’s not comparable?

Here are two complementary approaches to measuring development team productivity. I recommend using both of them jointly.

1. Accepted versus committed stories. The first approach is to get the high level assessment from the team as to which stories will make it into the sprint. Remind your teams that the goal is not to evaluate them but rather to discover if their productivity is as? they believe it is and in case they discover that it’s not, take corrective actions. Also, remind them that there is no requirement to always achieve 100% of the committed stories. Any number between 80%-120% should be acceptable (or whatever number you decide). Encourage the team to take some risks. You may be wondering whether counting relative stories is a good metric because some stories are bigger than others. I think that by keeping this metric as is you encourage teams to split large user stories which is a desired practice. For this reason, I’d stay with the ratio of story count.There is also a scaled version of this metric. At a group level or even the entire R&D level you can commit to content that is bigger than stories and takes longer than a sprint. However, in favor of clarity I will not get into the details of that in this article.
2. Delivered value rate compared to market need. If you satisfy only with the first metric, then teams can deliver all their committed stories and still the company can fail if for some reason the market requires a much higher delivery rate of product value. If your company needs to be able to excel in delivering a specific type of capability then it would be smart to decide on a target delivery rate for this.? As an example, think of a company that sells a smartphone app. Their app needs to be certified with a new version of smartphone OS immediately after the new OS version is published. If the engineering division can complete one certification per week but on average there are 3 new OS versions every 2 weeks then obviously the actual certification rate is under the needed rate. Another example can be taken from the automotive industry. Every country has specific regulations that every car? needs to adhere to. When a new car model is created, car manufacturers want the new car model to successfully pass country regulations very quickly otherwise they will lose time in fighting market share. Think which type of projects your company needs you to repeatedly deliver and decide with the management team on a required delivery rate for this capability. Then, measure the actual delivery rate and see if you are achieving your goal or not. If not, this can be a good input for discussion with your CEO on the investment needed to achieve your goal.

Make Metrics Visible and Present

If you decide to implement metrics, employees need to have easy access to them and they should be continuously in discussion.

Create dashboards that are accessible via browser. Hang screens presenting important metrics and status so that people can converse about them and take actions. By doing that you raise awareness to the existence of metrics and you remind people what success means in the organization. For this reason it is important to not overwhelm with metrics. Make it simple and easy to align with.

Data for the KPIs can come from various systems and tools .For example, build times can be taken from CI/CD tools, acceptance rate and aging bugs can be taken from your project management tool, build times on developers’ development environment can be measured and reported to a central location automatically as part of the build procedure. If you want to get high quality data for your metrics you’d need to invest in order to record this data. This is yet another reason that you should keep it simple and at every point in time settle for not too many metrics.

Reflect

Metrics are not useful unless you take time to reflect on them. You should decide how, when and in what forum. Some metrics can be discussed during weekly meetings with managers while others can be discussed quarterly. There are several things to consider during these meetings:

1. Keep, Adjust, Remove: decide if the metric is doing a good service for the company. Is there anything to change or try? Decide if you want to keep it, change it or abandon it
2. Act. Metrics can provide warning signs or call for change. Prepare a plan to drive improvement. Plan how you can score better next time
3. Celebrate success. When KPIs show improvement, celebrate success with the team. This will show everyone that metrics matter and will encourage them to pursue further improvement

Summary

Metrics are a great tool for steering improvements. Make sure to choose your metrics smartly so that they serve organizational goals. Insert metrics into the development process and make them visible, accessible and useful. From time to time reflect on your metrics and decide what actions to take.

I’d be happy to know if you found this article useful and if there are any insights on metrics that you learned over time and care to share.