Measuring Developer Productivity

It is hard to believe it, but we currently don’t have a singular effective and accurate way to measure developer productivity. There is some research and frameworks (detailed below) that seem to be giving leadership some idea of productivity within their organization, on the team level, but even those don’t allow to pinpoint relative productivity of individual developers. Why is it so complicated and what is a CIO to do with this? At the end of the day, some indication of strengths and weaknesses is required for larger IT organizations to gauge efficiency.

The easiest way to understand the complexity of this challenge is to examine it from a delivery point of view.

What do developers do (most of the time)?

Write code.

What is code?

A set of machine instructions that produce some outcome based on a given input. Input can be in form of a message, file or user request. Outcome is usually some persistent data and/or input to another program.

Then why not evaluate developers by the number of lines of code they write and/or average coding days? In this example we can define an “active coding day” as when a developer achieves some number of committed changes.

Unfortunately, this approach does not work. Once developers recognize this method is being used as a measure of their effectiveness, they can easily “play” the system to make them each look like 10x Developers (mythical creatures with super productivity). You can think of it in terms of a “Schr?dinger's cat” experiment. The very act of trying to measure some behavior (in this case development process) changes the characteristics of the outcome.?Developers will start writing much more verbose code with elaborate routines just to be considered “good”. Others might get their code commit numbers through changing comments, updating spelling mistakes in README files or updating irrelevant parts of the code – there are endless possibilities.

“Measuring programming progress by lines of code?is like measuring aircraft building progress by weight.” ― Bill Gates

So coding is not it.?Code, in and of itself, is not a product.?What you deliver to the client is a true indication of progress. This is where DORA comes in—DevOps Research and Assessment metrics.?DORA methodology factors in the following:

• Deployment Frequency,?Mean Lead Time for Change?(time taken from requirement being raised to the code being delivered).
• Change Failure Rate?(how often things break).
• Time to Restore Service?(if something breaks how long it usually takes to restore it).?

These indicators cover all aspects of development.?Depending on the nature of the application, team composition and architectural decisions the numbers will vary, but over a longer period of time if there is a team that has better DORA metrics than others, they can be considered more productive.

Why is not everyone jumping on this? First, it is not easy to obtain accurate metrics for these categories. You must be able to have complete traceability between multiple systems that are used for requirements gathering, source code management, production deployment and runtime infrastructure. Secondly, applications go through different lifecycles that can severely affect DORA’s individual metrics between sprints–i.e., delivery intervals. It doesn’t mean a team became less productive. Maybe they are researching the solution to a complex problem or helping clients with documentation. It doesn’t mean their productivity worsened. You can find more info on DORA here.

Finally, this is still on the team level.?One team might be composed of senior developers, business analysts, quality engineers, etc. Another team may still be solid, but not as experienced. It will be hard for the second team to compete solely based on DORA numbers, as the overall seniority of the team plays a key factor in the delivery.

Another popular measurement framework is SPACE. It tries to measure developer productivity based on mostly softer factors:

• Satisfaction and wellbeing
• Performance
• Activity
• Communication and collaboration
• Efficiency and flow.

These are good indicators of productivity. Details of how this framework works are beyond this post but it is also very subjective. Once the subject (i.e. developer) realizes SPACE would be applied to measure effectiveness, they could quickly change the parameters of their day-to-day activities to make themselves appear as the perfect employee. Original source paper on SPACE can be found here.

The latest trend for measuring developer productivity is the most direct one—ask developers how productive they are through a survey (you can check out this publication for more details).?Surveys are gaining more popularity as a tool to find a solution or recommendation to given problem. Answers to individual questions, combined with responses from other questions with industry baseline, can provide some insight to the problem at hand. Each of these surveys is usually backed by some heavy research and may come with a price tag.?The issue that many companies are facing with this category is the response itself.?You send too many surveys, and your response rate drops. If it is too long or too complex, the audience will simply breeze through it to be back to their favorite actively (which might be coding). If you must push people to complete those surveys, you will never receive genuine answers (back to?Schr?dinger's cat). Response rate below 50% should also give you some indication of a systematic problem.

Then come the “side” tasks. The hardest activity to measure. While researching and writing this article I was not really delivering any value to my employer. What is the empirical value of this article? I’m hoping that some number of readers will be able to learn things from it and become better managers. The desired outcome is not measurable.?

What is the value of deleted code??One such change on the existing codebase may require hours of testing, debugging and verifying impact. If I add a missing test, it is as important or less important than the production code. There will never be a metric to capture those nuances as development is art and true beauty can’t be measured by numbers.

Since I try to make my blogs practical here is your call to action— leverage all mechanisms above to get a general sense of current productivity levels and gaps. Start by trying to measure average number of code commits per day to identify areas that might be below some baseline. Spend the time with those teams to understand what drives them and apply necessary fixes to their challenges. The challenges holding them back might not be related to the technical excellence of the team or individual, but rather related to a plethora of other organizational impediments. While this is happening, try to get DORA setup. This will expose another set of rocks in your lake. Help to address those. When all said is done, send a survey to developers asking them what else they need to make them more productive and continuously improve upon the experience. Developers need the right support in order to be productive, so I challenge all engineering leaders to take initiative that goes beyond just measurement.

?

“All opinions are my own and not affiliated to my employer.”