How AI Code Assist Tools Create Value

Before we can know if a new tool or practice or process is helping we have to anticipate what advantage or leverage it is expected to bring to individuals, processes, or the org at large.

The early controlled studies like the GH 55% study and the McKinsey study, along with reports from many customers, show that "dev task time savings" is the most reliable benefit of using this technology. Dev Task Quality or Costs can also be improved sometimes. For example, a recent GitHub study on quality found that when developers are expected to improve quality and were given a way to gauge quality, they can then use GitHub Copilot to deliver higher quality code.

It's a common mistake for companies to try to measure the time savings benefit using process cycle metrics. Even though this approach occasionally reflects a shadow of "dev task time savings", in the majority of cases I've seen, it doesn't.

If it works sometimes, Why is this a mistake?

The main reason is that AI Code assist tools are directly impacting task efficiency as a first-order impact. There's no direct impact on process level efficiency. This means task duration, and consequently process cycle time, are only "contingent" second order impacts that can easily fluctuate. For example, Devs are able to use time savings in different ways. If they are empowered to use their judgment they can:

1. Intentionally shorten task duration
2. Intentionally improve task quality (testing, test coverage, documentation, or modularity)
3. Intentionally try to do more work within given task duration
4. They can even use the time for something unrelated to the task like learning, experimentation, work-life flexibility, or knowledge sharing.?

No telemetry can capture these micro decisions, or the varying amounts of time-saved routed to various intentions.

( Note: organizations that insist on glossing over these decisions are likely to fail at increasing the level of decision-making and decision-quality that will be required to benefit organizationally from improving Dev Task Efficiency. )

Improving Task Efficiency will often create pressure to increase Cycle frequency, but doesn't necessarily improve cycle overhead. If the time freed up by more efficient tasks is used to create a more efficient process, or to remove inefficiencies, then subsequent cycles will have less overhead - this is a path of compounding returns on time saved.

If, on the other hand, time savings merely is used to start the next cycle sooner, then that perpetuates the same cycle overhead or even increases it. This scenario is a path of diminishing returns on time saved. In this path, whether there is a net improvement in "productivity" entirely depends on whether there is enough Task efficiency gain to offset the increase in Process overhead that comes from more cycles. This is a path I heavily discourage because it will lead to muted benefit in the short term, and is unsustainable in the long term.

The above diagram walks through the "domino effect" that AI Code Assistants can have at the Task, Developer, and Org Level. In future articles, we'll explore what this model of improvement means for Measurement, Value potential, and why we must improve Decision-Making to keep making progress.