Concrete measure for eng efficiency

My last post “Clive’s Law” was a way to think about developer efficiency for a whole organization.

As I mentioned in another post I never had a CEO that didn’t ask me how we could make the engineers more efficient. My response was invariably, “how do we know how efficient they are now?”??

What is a concrete measure for an individual engineer??

This is the holy grail leaders have been searching for since we began coding. Back in the 70s there was Lines of Code (LOC).? LOC could be easily gamed, though really became meaningless when code generators started expanding checkins.

I remember an engineer in my first company proudly telling me they had "written 200k lines of code this week".

A break through?

During a stint at a larger company we uncovered something that gets closer to a metric.? It was enough to point out to my team's managers that someone was not being recognized for their efforts. She was immediately put on the promotion path as a result.

Here is the formula we used for ranking (Over some meaningful time 6-12 months)

Total number of Checkins * 80% + Total number of Code Reviews * 20%

This total for each engineer gave us an exact match for the stack ranking and other qualitative inputs that managers try to use as a proxy for efficient engineers.

Better though it gave us concrete numbers with large differences in totals not visible to a simple stack ranking.? The top engineer really was totaling 3-4 times higher than the next.?

In our data it also identified one outlier, who as I mentioned above, should have, on further discussion, been higher in the rankings. This total helped to show their manager why they had been underestimating her impact relative to the engineering in other teams.

It turns out that the best engineers checkin more times and review more checkins than their peers, over a reasonable period of time.

Concrete numbers at last?

This was the first time I had ever seen anything come close to a concrete way to compare engineers relative efficiency and value.

Of course it comes with a number of caveats.?

• It depends on your process
• It depends the product
• It only works for engineers who are full time coders*
• Engineers many have other qualities the team needs, like XFN communication skills
• Vacations, illness, and other large periods of absence
• etc

*? More senior engineers typically have other duties such as overall system design that take away from simply being coding machines.

Does it have Value?

This number is just one input to an overall review and management process. I have used it in the past to identify outliers in a positive way.? Managers use tools like the 9 square assessment and stack ranked lists of engineering teams in an effort to calibrate performance across a given team and multiple teams.?

A metric like this does have value, though use with caution. It can’t be used as the sole input.

It can also be used as a directional indicator for the development team as a whole.? If the overall total are decreasing over time it may be an indication that the development process has become too heavy and is slowing engineers down. A recent example of this have been the processes that have had to be put in place to address privacy regulations.?

It is also helpful in identifying teams that are using different processes.? Perhaps there is some knowledge that can be shared across teams.

Gaming the system

As with any metric it can be gamed. The way to game this one though is to checkin more often and review more of others checkins.? Unless the number of checkins is ridiculous, more checkin and more reviews is behavior we likely want to encourage.

My reservation of telling the engineers about this metric was the negative effect it might have on their work.? The pressure to checkin might prove a negative to quality or push engineers to take on simple projects where longer periods of design or experimentation were not necessary.? This is part of the reason the metric can only be one of many considerations in evaluating performance.

Is it worth the effort to collect this data

Most modern source control and build processes make the availability of this data trivially easy to produce.?

Give it a try and see how it correlates with your qualitative feel for engineering efficiency. Across teams and within your teams.

You may have to tweak the % weights depending on your process and your mileage may vary. I would though be very interested to hear your results in the comments.