GitHub Copilot - Key Factors for Measuring Productivity!

When discussing GitHub Copilot and its advantages for programmers, the pivotal question often arises: "How does it benefit my business?" Ultimately, technology proves its worth in business by either driving revenue or cutting costs (i.e., enhancing productivity).

In this post, I will propose a metric based approach to measure productivity of a programmer or a team who are using GitHub Copilot.

Why Metric Approach?

Frequently, there's a challenge in quantifying the impact of a technology on a business's efficiency or revenue. This can result in a nebulous understanding or rationale regarding the success or failure of technology adoption, particularly when non-technical leaders inquire, and you're tasked with justifying the technology.

The optimal approach to sidestep this predicament is to devise a method to attempt and calculate the business metric using available data from the technology, whether measured directly or indirectly. Fortunately, this task has become simpler with the majority of technologies offering avenues to share statistics on adoption, usage, and more.

Business Metric for GitHub Copilot

When it comes to GitHub Copilot, two business KPIs that can be directly observed for impact are the Sprint Velocity and User stories lead time.

The following information is derived from the latest API provided by GitHub Copilot, offering enterprise/organization-level statistics.

Prior to delving into the formula, let's familiarize ourselves with a few key terms and methods of measurement.

LoC Generated

This metric quantifies the number of lines of code (LoC) generated by GitHub Copilot.

No. of Prompts

A record of the number of prompts used to generate lines of code (LoC); fewer prompts for maximum LoC demonstrates efficiency.

LoC Accepted

Of the lines of code generated by GitHub Copilot, how many were ultimately accepted by the programmer?

Time Gained

Quantifying the time saved by developers due to generated/accepted lines of code (LoC) can be challenging. Initially, we rely on developers' input to gauge this metric.

Time Extension

Typically, after GitHub Copilot generates the code, programmers often need to invest effort in either refining it or making slight edits to align with the overall project code.

Technology Index

A lower-priority metric that assigns a score to indicate the ease of coding in a particular programming language. It takes into account factors such as the learning curve for programmers, the availability of talent, and industry adoption.

Programmer Level

A gauge of developers' skill sets, taking into account more than just the number of years of experience.

Calculating the Productivity

GitHub Copilot Efficiency (GCE)

This is a directly measure of how efficient GitHub Copilot is with your own context and code as a reference.

GCE = LoC Accepted / LoC Generated

Programmer Productivity

Two additional factors that you need to ger productivity is based on your current team skills and releases are the:

• Average Minutes to write single LoC.
• Time in Minutes to write a prompt.

Based on above factors and metrics listed, a Productivity formula would look something like this

Programmer Productivity = 
(LoC Accepted* Avg Minutes to write LoC) - (Time Gained) 
/ 
(Time Gained)        

As i mentioned on top, above calculations can be done using latest API released by GitHub Copilot team which returns the metrics at GitHub enterprise or organization level.

Conclusion

This is an initial thought based on my own efforts and eagerness to learn whether it has helped. This needs to be undergo multiple iterations to finetune and settle with a metric and a way to calculate the same.

Will keep updated on the updated version of the calculation in future posts.

If you have better idea to calculate, DM to connect and collaborate!