Agile Efficiency Metrics: A Cautionary Tale

The Scrum Guide establishes the methodology’s origins in empiricism: 

“Scrum is founded on empirical process control theory, or empiricism. Empiricism asserts that knowledge comes from experience and making decisions based on what is known.“

The methodology advocates measuring team performance and applying improvement strategies (inspect and adapt) as the team continuous improves (kaizen). It’s important for Scrum Masters (SM)  to select efficiency metrics that match a team’s maturity level. New teams need to master decomposing epics into stories, estimating stories and swarming on the highest priority stories to achieve the Definition of Done. Velocity and accuracy are the primary metrics at this level. A team has reached the “forming” level when the Product Owner (PO) consistently accepts a total number of story points greater than or equal to the value the team selected during sprint planning. Such a team can be labelled “predictable” based on the measurements taken at the sprint boundaries.  The team can now pursue the next level, “efficient,” by measuring their performance within a sprint via cycle time and efficiency metrics. 

During the next evolutionary step, I encourage SMs to approach the efficiency topic from the team’s perspective rather than from a metrics perspective to minimize a defensive reaction from the team. I’ve seen several negative repercussions from taking the latter approach. For example, an executive decides to measure user story cycle time across an organization. Each SM is tasked with collecting the data for quarterly reviews. Teams may voice the following complaints:

• “Why does it matter how quickly we complete each story? We’re completing the stories our PO prioritized within the sprint. Why are we being pushed to be more ‘efficient?’”
•  “Management will use the efficiency metrics to compare teams and to take punitive actions with insufficient context. Teams work on different types of stories and use different strategies for sizing their stories. Therefore, you can’t compare teams. Most teams have “rough” sprints occasionally due to events outside their control (illness, customer escalations, infrastructure issues). Management won’t consider the context before drawing conclusions.”
• “The efficiency numbers are vanity metrics to make management feel good. We can easily turn the process into a “numbers game” by delaying when we mark stories In Progress to produce more favorable metrics. But such tactics are sources of toil that don’t benefit us or the customer.”

Visualize the Work

Rather than broaching the efficiency conversation by presenting a set of numbers, especially if the numbers indicate the team is struggling, I encourage SMs to depict recent sprints as a function of each story’s duration (business days from when the story transitioned to In Progress to when it transitioned to Done) and the story size. The depiction should also illustrate how many stories were in progress at the same time. Below is an example graph.

During the sprint review ceremony, the SM can pick a story with a high duration-to-size ratio and ask the team to reflect on their experience completing the story. 

• Did the story involve multiple dependencies and wait time? 
• Did the story’s Definition of Done evolve during the sprint? 
• Did team members work on the story the entire time it was “in progress” or did they get blocked or interrupted? When a blockage occurred, did they move onto another story, thus increasing the Work in Progress?

If these or other scenarios occurred, how did team members respond? Were they frustrated by another team’s late deliverable? Were they annoyed by a stakeholder interrupting their sprint with an “urgent” task that should have been processed during sprint planning? Once the SM has identified pain points from the team’s perspective, he/she can explore strategies for reducing the team’s frustration. Perhaps the team can take ownership of a deliverable that typically arrives late and thus break the dependency with another team. Perhaps the PO can collect potential stories from stakeholders prior to sprint planning to minimize mid-sprint interruptions. The team can craft improvement stories around these strategies for future sprints.

Servant Leadership

The scenario described above has the SM acting as a servant leader—understanding what issues impair the team’s performance and then working to minimize these impediments. This approach builds trust through the Scrum principle of “individuals and interactions.” The team sees the SM and stakeholders acting in the team’s interest. The unspoken by-product of this approach is that efficiency metrics likely improve because the team experiences fewer impediments and interruptions. The team’s happiness metric also likely improves because its frustration level decreases. The SM can continue to “visualize” the work so that the team sees the improvement from their perspective not just as a cycle time metric tracked by an executive. As the team passes through the “efficiency” stage, the efficiency metrics will likely stop being a hot button issue because the team has experienced the benefits of working efficiently.