A playbook to school LLM Agents to create your AI-Powered knowledge workforce:

Imagine your ideal entry-level consultant: a research whiz, a data visualization maestro, and a tireless report-generating machine. Now imagine this same individual lacking the experience to navigate complex problems, identify crucial information, or build credibility with clients. If only if you could quickly show them the 'ropes' of how you do business, what amazing value you could unlock with this 'super-associate'.

That is the opportunity presented by agents powered by Large Language Models (LLMs). These AI marvels boast vast knowledge bases and exceptional work ethic, capable of research, report generation, and even coding – the ultimate entry-level dream team.? Consulting firms or really any organization in the knowledge & content economy that leverage LLMs can gain a huge competitive advantage by deploying effective LLM powered agents as ‘armies of super associates’.

?But similar to a fledgling consultant, LLMs require guidance to become truly impactful contributors.? Breaking it down further, these are some gaps that both LLMs and entry-level consultants share:

?Brilliant But Inexperienced: The LLM Conundrum

• Domain Expertise: Applying mountains of information requires understanding the context. LLMs and entry level associates lack this, needing direction on how to leverage their knowledge for real-world scenarios.
• Information Discernment: Sifting through a sea of data and identifying the most relevant pieces is crucial. LLMs and new employees, without proper guidance, can drown in information overload (or worse make up information)
• Knowing what good looks like: LLMs like associates need a standard of what good looks based on prior work that they can work towards
• Building Credibility: Past successes build trust with stakeholders. As the ‘new kids on the block’, LLMs, lacking a history of accomplishment, require outputs to be vetted for accuracy before wider use.

?Building the LLM Dream Team: A Business-Engineering Partnership:

Effectively harnessing LLM potential requires an intentional partnership between business and engineering sides of the organization to translate business logic in to AI pipelines. As we have gotten practice building LLM fueled applications and processes, we are now starting to develop a high-level ‘playbook’ for resources we can use to ‘up-skill’ an LLMs that mirrors the resources we give new consultants (& all the attendant challenges of knowledge management).

• The Training Manual - Prompt Engineering:

Just as we provide clear instructions to new hires, "prompt engineering" defines how we ask questions of LLMs. This involves crafting detailed, written instructions that guide LLMs towards the desired outcome. Think of it as the training manual for your LLM associate. Like any good training manual, the process starts with as ‘business’ manager who can define a process that works across diverse questions with an added challenge (& opportunity) for engineering to translate that defined process in to instructions to an LLM.

• The Reference Library- Retrieval Augmented Generation (RAG):

Remember the mountains of uncontextualized information overwhelming a new associate? We curate relevant data for them. Similarly, organizations need to establish "RAG" systems. These systems point LLMs to domain-specific knowledge and data for them to access and leverage. This is their reference library, ensuring they have the right information at their fingertips. As with any reference library, it is often only as good as the catalog that describes the library assets and accompanying instructions on how they can best accessed and used. Great RAG based systems are a collaboration of the business and engineering sides to curate the right data along with descriptive meta-data made accessible to the LLMs in right format and infrastructure.

• Specialized Training Programs- Fine-Tuning:

Formal training programs are critical for consultants, which means not only creating reference materials but taking the time and effort to show them how to use the materials as well a time to practice. For LLMs, "fine-tuning" involves exposing them not only to relevant data but also required outputs labelled based on their quality and the ‘practice time’ to be able to create the high quality outputs independently. Like specialized trainings, fine-tuning is expensive, time consuming and requires specialized people and resources for the average business to execute (today) and is something to be reserved for specialized high-value use-cases

• Mentorship & Quality Control – Human Review & Workflow Citations:

No self-respecting manager lets a client (or even project partner), see an associates work without review and usually a few iterations. Similarly LLM outputs today require human review and the willingness to go through multiple drafts/prompts to get a ‘client-ready’ output. Building citations and workflow audits are an essential component of LLM powered processes because unlike with an associate, you cannot call an LLM to have them show you their work. Ofcourse neither the best associate not the best tuned LLM is a substitute for a diligent manager or human review. Like managers and associates, LLMs produce their best output in partnership with humans not left alone on an island.?

The LLM Advantage: An Essential Competitive Edge in the Knowledge Economy:

Using LLM’s effectively means an ‘up-skilling’ process that requires organizations to translate their day to day business activities in to a structured set of steps that can be encoded as instructions to an LLM. Organizations that foster a tight partnership between business and AI engineering teams are the ones best positioned able to execute LLM up-skilling. In the knowledge and content economy, those who effectively use playbook elements to leverage LLMs gain a significant edge since? they essentially have an army of tireless, cost-effective associates at their disposal. And those who do not are working with a hand tied behind their back.?