Measuring Engineering Productivity

Each week in the SE Research newsletter, I share a summary of a research paper relevant to understanding and improving developer productivity.

This week I read?Measuring Engineering Productivity?by engineering productivity researcher Ciera Jaspan . This is a chapter from the book?Software Engineering at Google?that describes the rationale for measuring productivity, as well as Google’s framework for selecting metrics.?

My summary

Why measure at all??

Before choosing metrics, first understand your reasons for measuring. Jaspan opens the chapter with a simple explanation: one reason for measuring is to scale the scope of the business by making engineering more productive, rather than by hiring more engineers. The goal of metrics is then to identify inefficiencies in engineering processes and decide which areas to invest in.?

The rest of the chapter outlines a process for selecting meaningful metrics.

Selecting meaningful metrics

Google uses the Goals/Signals/Metrics (GSM) framework to guide metric creation.?

1. Start by defining the goal you’re trying to accomplish.?Your goal should be phrased in terms of what you want to understand at a high level and should not contain references to specific ways to measure it.?

Aim to cover different parts of productivity with goals. Jaspan explains that Google divides productivity into five components, which they refer to with the mnemonic “QUANTS”:?Quality of the code,?Attention from engineers, Intellectual complexity,?Tempo and velocity, and?Satisfaction. Each component has an associated goal. (See the table below for examples.)?

2. Define signals.?A signal is how we know if we’ve accomplished the goal. Every goal should have one or more signal. Not all signals are measurable.?

3. Select metrics.?These are proxies for the signal. Some signals might have multiple metrics.?

Selecting metrics also includes deciding how to measure them. In the example below, surveys and logs data are used. Jaspan writes, “Qualitative metrics are metrics, too! Consider having a survey mechanism for tracking longitudinal metrics about engineers’ beliefs. Qualitative metrics should also align with the quantitative metrics; if they do not, it is likely the quantitative metrics that are incorrect.”?

Following the GSM framework is a great way to clarify the goals for why you are measuring your software process and how it will actually be measured.?

Consider whether the metrics are worth capturing?

Jaspan also shares the questions Google uses to help determine whether these metrics are worth measuring:?

• If the data supports your expected result, what action will be taken??If no action will be taken, there is no point in capturing that metric.?
• If we get a negative result, will appropriate action be taken??If there are other inputs that will override a negative result, it might not be worth measuring.?
• Who is going to decide to take action on the result, and when would they do it??“The goal of measuring our software process is to help people make business decisions. It’s important to understand who that individual is, including what form of data convinces them.”?

Final thoughts

I recently read a?post by Will Larson?that articulates the common reasons for measuring engineering. Of the reasons Will discusses, today’s issue is most relevant for leaders looking to “measure to optimize,” or in other words measure to understand what needs to be done to improve productivity.?

The Goals/Signals/Metrics framework is an important tool that can help engineering teams get clear on what they want to understand and how they’ll use that information — before going down the rabbit hole with metrics.?

That’s it for this week! I'd love to know your thoughts in the comments below.?For more software engineering paper summaries, subscribe to the SE Research newsletter on LinkedIn or Substack.