Empowering Agile Transformations: Developing a Custom Agile Bot

In the ever-evolving world of Agile transformations, efficiency, innovation and scale are paramount. Tailored solutions are not just beneficial; they are essential. At Notch Above Consulting, we pride ourselves on leading the charge in Agile enablement. We actively collaborate with our customers throughout their Agile journey, ensuring our solutions meet their evolving needs and drive maximum value. This is why we've developed a unique Agile Bot designed to support these transformative initiatives. But why did we create our own bot when there are already powerful Large Language Models (LLMs) like GPT3.5-turbo, GPT4, GPT4-turbo, Mixtral 8x7b, Llama3 8b & 70b??

This is a great question!? LLMs like GPT-4 are indeed powerful, capable of retrieving data from countless sources across the internet. However, using those models alone doesn't guarantee the best practices or relevant information tailored to a specific organization's needs. Agile teams often look for specific information including team makeup and capacity, dependencies across teams, meeting schedule for stand-ups, sprint reviews and retrospectives, product roadmap, lessons learned from previous program increments and more. This is where the concept of an internal Agile knowledge base becomes crucial.?

Our internal Agile knowledge base at Notch Above Consulting is a curated collection of documents we've developed through extensive hands-on experience with clients. Unlike general Agile documents available online, our corpus is finely tuned and highly specific. It addresses the challenges and issues that we have observed during our customers' Agile rollouts. One significant challenge is organizing, managing and retrieving data. Examples include Agile training and onboarding resources, role definition and assignments, coaching and support information, toolkits and templates, roadmaps, success criteria, KPI’s, team information, program implementation planning schedules, scrum processes, and reports.?

We built our internal Agile knowledge base through diligent and comprehensive hands-on work with our clients. This involved collecting data, insights, and experiences from various Agile implementations. These documents were then refined to be highly relevant and practical, ensuring they met the unique needs of our clients.?

Because each client is different, with varying levels of Agile maturity, toolsets, team structures and product offerings, our approach begins with understanding each client's specific environment. This allows us to integrate this context into our knowledge base, making the content highly relevant and unique for their specific Agile transformation journey.?

To enhance the intelligence and responsiveness of LLMs using our internal knowledge base, we employ a Retrieval-Augmented Generation (RAG) pipeline. Here's how it works:?

1. Knowledge Base Integration: We load our specific, fine-tuned Agile documents into the system.?

1. Content Retrieval: When a query is made, the system retrieves relevant content from the knowledge base repository.?

1. Response Generation: The retrieved content informs the LLM's response, ensuring it is accurate and contextually correct.?

The next phase involves effective content chunking which is crucial for maintaining contextual relevance. We tested various models. By optimizing content chunking through a blend of scientific and artistic methods, and retesting, we ensured that the indexed data remained coherent and contextually accurate. Fine-tuning these chunks has significantly improved response accuracy and relevance.?

Lastly, roles are critical in Agile, so we have included specific Agile-related personas in our knowledge base. This helped ensure that the responses are tailored to the roles and responsibilities of different team members, enhancing the practical utility of the bot.?

We even tested the recently released LLM Llama3 8b and 70b models by Meta.??

Our metrics indicate substantial improvements with fine-tuned content chunking and persona layer:?

• Initial content chunking accuracy: 70%?
• Post-optimization chunking accuracy and keywords: 90%?
• Persona and role-specific relevancy and accuracy: 95%?

The integration of RAG with an internal Agile knowledge base represents a significant advancement in delivering instant, customized, actionable insights. By combining the strengths of LLMs with domain-specific knowledge, and in the midst of lots of moving pieces, we provide unparalleled support for Agile transformations, ensuring our clients receive the most relevant and practical guidance tailored to their unique needs.??

Imagine a new team member having instant access to a wealth of knowledge specific to their scrum team. By offering highly relevant content and specific responses, we empower teams to navigate their Agile journey efficiently and effectively, thereby increasing the organization’s overall productivity, adaptability, and success in achieving strategic objectives. With our solution, you can expect to see reduced onboarding time, less carryover stories into the backlog and a measurable boost in project outcomes. Additionally, our tool supports a culture of continuous improvement and collaboration, enhances transparency across teams, and ensures alignment with Agile best practices. As a result, your organization can anticipate not only achieving faster delivery times but also experiencing greater employee satisfaction and retention, ultimately driving long-term growth and innovation.??

To leverage the transformative power of our solution and propel your organization towards Agile excellence, take the next step today. Schedule a demo to witness firsthand how our tool can utilize your team's productivity and drive unparalleled results!?

Your perspectives are vital to us, and we invite you to enrich our collaborative learning ecosystem with your feedback. Connect with like-minded innovators and contribute to the dialogue by joining our community here .??

Authors:?Monica Connolly ?and?Jai Natarajan ?

Monica Connolly is a Senior IT Program Manager at Notch Above Consulting. Monica is a certified SAFe?Agilist supporting the internal AI?development team. Jai Natarajan?is a Managing Director at Notch Above Consulting and a passionate advocate for the transformative power of AI.?