The key KPI for AI Assistants - Time to Proficiency

Today, some companies are struggling to define the real value of AI for their organization. In many ways, we have reached the peak of the hype cycle. AI shows promise, but few organizations can demonstrate measurable results. The promise is still there, thought. I believe that companies harnessing AI can demonstrate its tangible value and their commitment to meaningful digital transformation.

At the top of the hype cycle is the AI assistant. Tools like ChatGPT and Copilot are widely used in organizations (even if their use is not always acknowledged). This broad use provides a valuable opportunity to demonstrate the value of AI.

There are several ways to evaluate the benefits of AI assistants. While much of the literature focuses on the time saved through search, there is a better lens through which we can see the real benefits of these assistants. The best candidate for measuring AI assistant value is time to proficiency. I work in the construction industry, and there is a significant gap in the number of qualified workers and the need (we need many more). Given that gap, reducing the time that it takes to become a truly productive contributor could be a significant advantage for a company.

When looking at the evolution of AI assistants, there are three moments that we believe will lead to a reduction in time to proficiency:

1. Retrieve – find project information easier (key value, time savings)
2. Assist – data points to suggestions, like triggering a workflow action or creating inspections under certain conditions (key value, time savings, and reinforcing best practices)
3. Guide – suggest best practices under certain conditions (key value, time savings, and time to proficiency)

The open question is how to define competence. I would suggest that it is not defined as not only able to work independently, but also capable of working in a way aligned to the way the company does.

?

I am starting to measure the ways that AI assistants could reduce time to proficiency, creating a baseline as we adopt and enhance AI assistants. As these assistants evolve, I believe they will have a much more meaningful impact. They can substantially reduce the time to proficiency (and thriving) for new team members. However, we cannot make significant investments without building KPIs that demonstrate their value.