AI's Agile Breakthrough: Introducing an AI-Driven Assistant for Superior Coaching

Improving agile planning and forecasting with AI and an easy-to-use interaction.

Co-authored with AI expert and LinkedIn Top Voice Tobias Zwingmann , author of "Ai-powered Business Intelligence: Improving Forecasts and Decision Making With Machine Learning" O’Reilly

Hey there,

This week, we're going to look under the hood of the most sophisticated agile tool we've ever wanted.

We are not called for nothing. Something is burning. Organizations often face the same problems. There is a lack of overview. There is a lack of focus. And there is a lack of a system that enables the teams to solve these problems.

Recurring problems, mountains of data in ticket systems, and simulations of the future sound like tasks with automation potential. While every organization is unique, there is no need to reinvent the wheel.

That's why I've teamed up with AI expert Tobias Zwingmann to create Agility GPT, an assistant that empowers teams to better understand their system and develop their own solutions.

But what can it do? AI analyzes your Jira data to help you turn insights into action. For example, Agility GPT helps you better understand the flow of work to make better predictions. Something that is really hard to do. Some people even say it can't be done.

We are convinced otherwise, so let's find out!

Agility GPT is not yet available to the public. But as a newsletter subscriber you can claim exclusive early access TODAY. Just ping me with "Agility GPT" and I'll send you the link!

Agile Projects Are a Bit Like the Weather

If you want to forecast any work or a project which is organized in an agile way, it’s a bit like trying to predict the weather for a camping trip - it's important, but not always easy.

Agile projects change a lot (hence the name!) and figuring out how long the development of a certain amount of stories, features or even epics will take can be really tricky. Also, every team works differently, and things can change quickly. This makes it difficult to plan accurately.

Despite these challenges, being good at forecasting is very important. Especially in large organizations where delays can cost millions every day, it is a must. It's also key to keeping what you promise to (internal) customers about when you're going to get it done.

To make it a litte more tangible, let’s consider a concrete use case.

Situation and Problem Statement

Imagine you’re responsible for an agile value stream improving an existing software platform. The product owner shows up in one of your meetings and says that you "definitely” must include that super cool feature he’s heard from a competitor (did I hear someone say AI?). At the same time it is - of course - critical that the next release will be shipped as planned. So what do we do?

In agile value streams, we often use "story points" to estimate how much work a feature will take. Typically, the team comes up with a certain number of story points using a method like planning poker (the agile crew has even cooler words than the AI people).

Let’s say the team agrees that this feature in total would be about 200 story points.

You would assume that more story points (=more complexity and other dimensions) correlates with the time it takes to complete the amount of work (= cycle time). Unfortunately, this is rarely true in practice and that’s why forecasting is so hard.

To solve this, the solution would be to shift through historic data and perform an in-depth analysis combined with a proven forecasting methodology to inform planning. However, since they have rarely the time, and sometimes even rarely the skills, to do this, what happens is that they often fall back to using the story points as a time estimate. Coupled with an over-reliance on past performance this ultimately leads to poor forecasts.

That’s where I’ve partnered up with AI expert Tobias Zwingmann , to create the world’s first Agile Forecasting GPT to help solve these tasks.

And here’s how it works:

Solution Overview

Agility GPT lives (currently) behind the ChatGPT interface which means you need a ChatGPT plus subscription to use is. But looking forward, it could be as simple as a dedicated chatbot in Microsoft Teams or even Jira.

Agility GPT works by combining three core components:

• Detailed instructions to guide the user through the forecasting process
• A custom Python code that contains the forecasting logic
• Curated knowledge documents to provide more context

Here’s a high-level overview of the architecture:

There’s absolutely no technical knowledge required from the user! They only have to be a basic understanding in the domain of agile development.

Solution Breakdown

Let’s explore how this solution works by walking through the different stages.

Data Layer

To make it work the GPT needs some data. Right now, the easiest way is to take an export from a project management tool like Jira and upload this data as a CSV or Excel to the GPT. Mandatory columns are 'Issue Key' and 'Resolution Date', optional columns are 'Story Points', 'Start Progress', 'Issue Type', and others. Each row contains a distinct issue. This is a standard export format in Jira. Here’s how this could look like:

Looking forward, this could be simplified using a direct connection to the Jira API.

Upon uploading the data, the GPT will run some sanity checks and handle a pre-defined data preparation logic like removing duplicate issue IDs and handling issue IDs without resolution date, etc.

Analysis Layer

Within the analysis layer, the GPT can use the custom Python code along with instructions and knowledge documents to guide the user through the forecasting process. This includes three options:

1. Extended Data Analysis
2. Throughput Based Forecast (issues per time unit)
3. Velocity Based Forecast (story points per time unit)

Option 1: Extended Data Analysis

In this option, the GPT analyzes the historical throughput and velocity of completed issues from the forecast data set. This step allows the user to understand how throughput and velocity are correlated, if there are any historical outliers, and ultimately, which metric - throughput (issues resolved per time unit) or velocity (story points resolved per time unit) - is more appropriate for running the forecast.

Here’s how a Throughput vs. Velocity chart could look like:

Going further, Agility GPT will calculate the variation for both metrics and inform the user which one of both is better suited for the forecasting task:

Another way to look at this is the so called cycle time - the amount of time it takes to complete an issue. If story points were a good proxy for the forecast, there would be a high correlation between story points and cycle time. If that’s not the case, the GPT will point that out respectively.

After completing the data analysis (which takes about 5 minutes), the user will clearly understand if they should rather use velocity (story points) or throughput (e.g. user stories) for prediction.

The user can switch to the prediction-mode any time. Let’s switch gears as well and see how AgilityGPT performs the forecast

Option 2: Forecasting

First, the GPT asks the user for how many story points or issues they would like to predict the completion time.

Let’s take our example with 200 issues above. That could be stories, features or other issue types. The GPT will start with a static burn down chart forecast - well known to every agile coach and project manager.

The burn down chart can be used to estimate how long it will take to complete a certain number of user stories or story points based on average historic performance. Agility GPT does this and also explains the limitations of this approach:

The GPT runs a Monte Carlo simulation next, which considers not only the historic performance but also some variation in delivery pace and scope increase. As you can see, one burn down line now becomes 10,000 burndown lines, with a few 100 randomly plotted:

This will allow us to create a histogram of completion time from all simulations, giving us a better estimate and probability to when the work will actually be done.

As we can see, with 85% probability, the scope will be completed in 13 months or less.

Understanding Percentiles

• Probability Comparison: The 50th percentile, also known as the median is akin to a coin flip in terms of probability, meaning there's an equal chance (50%) of the actual outcome being either below or above this value. Obviously, planning based on the average cannot have an above-average probability of occurrence. That's why planning often fails.
• Usage: When more confidence is needed than a mere coin flip (50%), higher percentiles (like the 85th or 90th) are utilized to ensure a higher probability of the outcome falling below the forecasted time frame.

User layer

Since the user is interacting with a fully-fledged LLM, in this case GPT-4, the beauty is that they can ask any question they want about these plots! For example, if you struggle to interpret a particular plot or you’re confused about the relationship between cycle time and story points, simple ask the GPT and it will answer these questions based on curated expert knowledge an your own custom data.

To be clear - something like this was totally out of scope before the era of large language models. But now, it’s within close reach. While there’s definitely room for improvement, the GPT works already quite well.

Conclusion

Agility GPT improves agile planning and forecasting by integrating AI-powered forecasting and user-friendly interaction. By empowering users with insights such as cycle time, throughput, story point reliability, and scope completion estimates, Agility GPT is a groundbreaking solution for agile coaches, scrum masters, and project managers striving for delivery excellence and efficiency.

I hope you’ve enjoyed this deep dive into the world of agile forecasting. And don’t forget:

Send me a message with "Agility GPT" if you want access to the GPT and I'll send you the link!

Can’t wait to hear your feedback!

See you soon,

Frank