Will Productivity Metrics Sidetrack AI like Velocity Sidetracked Agile?

Having spent decades in the trenches of agile development, I’ve seen firsthand how the misuse of metrics like velocity can derail even the best intentions. In the early days of agile, we believed the movement would bring a new level of flexibility and customer-focused value to software development. Our goal wasn't just to ship code but to create something meaningful and valuable for customers. Yet, somewhere along the way, a key tool—velocity—morphed from a helpful indicator into a counterproductive target.

This shift led many teams to focus on inflating their numbers rather than delivering real value, sidetracking the agile movement. As we explore the role of AI in transforming our work, we must heed the lessons from agile to avoid repeating these mistakes. The risks are similar: allowing productivity metrics to hijack our focus and undermine AI’s true potential.

The ThoughtWorks Example: A Cautionary Tale

I remember a story from 2010 when I visited the ThoughtWorks office in London to consult with a client and meet with a team engaged in a project. After learning about their work, I asked, "So from your perspective, this appears to be a successful engagement. What is the client’s view?" The response was, "The client appears to be pleased with our progress so far." I then asked, "What seems to be the client’s greatest concern?" Without hesitation, the team replied, "Velocity."

This lesson from the past illustrates how a misplaced emphasis on velocity led agile teams to lose sight of what mattered most: delivering value to the customer. Similarly, AI initiatives today risk focusing too heavily on easily quantifiable metrics, leading us away from what truly matters—enhancing human capabilities and delivering genuine customer value.

I continued, "And what other metrics do you report?" The answer was, "None really, just velocity." This was a perfect example of how velocity, while useful for capacity planning, can become a poor performance metric that is counterproductive. To help the team refocus, I suggested they work with their customer’s product leader to assign value points—a simple relative 1-5 evaluation in addition to the estimated story points. At the end of each iteration, they could report they had delivered 35 value points and expended 25 story points. When I followed up a few months later, they reported that the customers liked the idea of value points and rarely complained about velocity anymore. Sometimes, a simple change in performance measures can have a profound effect.

Velocity is Killing Agility: A Systemic Problem

The issue with focusing on velocity wasn't limited to individual teams—it was pervasive across the agile community. Another story comes to mind: I was at lunch one day with staff from a prominent Agile software tool vendor, and one of them was touting their tool’s ability to roll up team velocity to the executive level. I was so aghast at this use of velocity that I wrote a blog post titled "Velocity is Killing Agility" (Highsmith 2013).

In the post, I asked the question: "Was James Michener a better writer than Ernest Hemingway because his books are longer? Or because he could type 20 words-per-minute faster?" No sense, right? Assessing the effectiveness of writers based on per-unit productivity (activity) measures makes no sense. Similarly, evaluating a team’s success based on velocity misses the point entirely. If your job is to write Java code—"words" rather than English words—similar productivity measures don't make sense either.

The systemic misuse of velocity as a performance metric showed how deeply ingrained this problem had become. AI today faces the same risk, using easily quantifiable productivity metrics to measure success at the expense of true impact. Productivity measures were originally meant for tangible outputs, like how many widgets a machine could manufacture in an hour. These measures were never designed to assess intangibles such as ideas and innovations. But measuring intangibles is hard, and measuring tangibles is easy, so people naturally gravitate to what is easiest—even when it’s wrong. Better to have some measure rather than nothing—right? No! Give me a fuzzy metric of something valuable (an outcome) rather than a precise metric for something unimportant (an output) any time.

AI at a Crossroads: Learning from Agile's Mistakes

We now find ourselves at a similar crossroads with AI. The promises of AI—augmenting human intelligence, solving complex problems, enhancing creativity—are profound and exciting. But if we aren't careful, we'll repeat the mistakes of agile by allowing productivity metrics to hijack our focus. Just as a misplaced emphasis on velocity sidetracked agile, productivity metrics risk narrowing the purpose of AI to mere output rather than its transformative potential.

Consider the current wave of AI models being integrated into knowledge work. There's a growing emphasis on metrics like the number of decisions automated, response times, and the volume of content generated. These metrics, while tangible, risk reducing AI to a productivity booster, undermining its true value. AI's real strength lies in supporting nuanced decisions, enhancing understanding, and helping us explore new possibilities—just as agile’s strength lies in adapting to change.

Instead of focusing purely on productivity, companies should prioritize delivering customer value through continuous innovation. The agile community has already learned, sometimes painfully, that focusing on the wrong metrics can lead to misguided efforts. We must avoid making the same mistake with AI, ensuring that our efforts are focused on outcomes that truly matter.

The Agile Triangle vs. the Iron Triangle

One of my primary goals for the second edition of Agile Project Management was to discuss performance measurement issues and introduce the Agile Triangle as a replacement for the Iron Triangle used in traditional project management. Just as the Agile Triangle shifted focus from rigid adherence to plans towards flexibility and value, we need a similar shift in our AI initiatives.

Agile teams often complained, "Management wants us to be agile and adaptive, but we also must conform to the project’s planned scope, schedule, and cost objectives." They wanted agility but were still measuring teams by traditional metrics. If adaptation and flexibility define agile projects, why measure their success using rigid traditional frameworks? Agility means adapting to inevitable changes, not rigidly adhering to initial plans.

The Agile Triangle addresses this by focusing on:

• Value goal: Deliver a product of value to the customer.
• Quality goal: Build a reliable, adaptable product or organization.
• Constraint goal: Achieve value and quality within acceptable constraints.

The Iron Triangle enforces conformance to static plans, whereas the Agile Triangle encourages adaptation to dynamic needs. Extending this concept to AI, we need to rethink how we measure success—not by how many tasks AI completes or how quickly it processes data, but by how well it enhances outcomes, enables better decisions, and fosters creativity. Just as the Agile Triangle encourages adaptation to dynamic needs, our approach to AI metrics must also prioritize outcomes that support innovation, adaptability, and genuine value creation.

Actionable Recommendations for AI Metrics

To avoid falling into the same trap with AI, we need to rethink our approach to metrics altogether. It’s crucial to focus on outcomes that truly reflect value rather than getting caught up in easily quantifiable but less meaningful measures. Here are some key areas to consider:

• Measure Innovation: Capture customer impact through innovation metrics. Use innovation storytelling to create narratives around how specific innovations have solved real customer problems. This approach highlights tangible improvements, providing a richer picture of success.
• Customer-Centric Metrics: Define metrics such as satisfaction scores, engagement levels, or customer feedback to keep focus on delivering genuine value. Customer-centric metrics help maintain focus on what matters most.
• Productivity Metrics in Context: Productivity metrics still have a role but must be used wisely. Customer value and adaptability always outweigh raw productivity. Improving productivity might deliver short-term gains, but it should never compromise adaptability and innovation in the face of rapid change—especially when driven by AI.

By focusing on simple, meaningful metrics, we can guide AI initiatives and agile projects toward real value creation rather than just chasing numbers. It’s about continuous learning, adapting, and ensuring that the work we do today lays the foundation for lasting success.

A Call to Action for Leaders

Leaders have an opportunity today to guide AI initiatives towards real value creation, just as we once aimed to guide agile projects. Let’s choose metrics that foster adaptability, creativity, and true customer value. By doing so, we can ensure that AI delivers on its promise—making our work richer, more human, and more meaningful.