The Trouble with Generative AI: Pt.2

Level:?Foundational? Reading time:?9mins

In Pt.1, we looked at the rise of GenAI, investment and costs, Machine Learning vs GenAI, Hallucinations and Semantic Search. I left you with the billion dollar question how does any organisation get real business value from GenAI?

Getting better results from LLMs

One of the initial shortcomings of GenAI outlined in Part.1, is its accuracy. Hallucinations can be significantly reduced with very little effort requiring zero programming skill. Writing better questions with more context, reduces the level ambiguity – enter Prompt Engineering.? The prompt engineering community have rapidly evolved a series of techniques to guide the LLM to the desired output.

Prompt Engineering Example:

...a well-engineered prompt which now contains context (through a persona), some input data, clear instruction and expected output format, eliciting a much more useful response.Often involving a lot of trial and error, more advanced prompt engineering techniques include:

Reflection (aka reflexion) and chain of thought (which forces reasoning) especially, are simple but surprisingly effective. A common and widely applicable use case for taking advantage of prompt engineering techniques via a programming interface, is the Conversational AI Chatbot. Traditional chatbots use keyword search, are rule-based and (let’s be honest) struggle to deliver a decent customer support experience. Conversational AI chatbots are capable of learning and delivering a far more dynamic one. By using semantic search, they can flex on tone and style to deliver more seamless conversation flow.

Prompt Engineering Business Implication: By employing Prompt Engineering techniques, we can reduce LLM hallucinations and increase the accuracy of responses.

LLM Fine-tuning

What if our question is outside the LLM’s realm of knowledge, either due to its cutoff date or the fact that your company’s internal knowledge bases and documents are confidential and therefore outside of the LLM’s original training dataset?

One option is to run a foundational LLM model on a cloud platform and finetune it with supplementary data about your organisation and its business domain(s). But there would be a considerable cost to hosting an LLM, from ongoing training to operational business use, not to mention the data science effort required from hard to find talent. It’s a hotly debated topic, but I believe model finetuning should be used when trying to change the behaviour of a model, not to add supplementary data.? There is a considerably quicker, cheaper and potentially no less effective route to achieving this…

LLM Fine-tuning Implication: This iterative process requires data science expertise alongside considerable on-going model training budget - LLM fine-tuning should be seen as a last resort.

Don’t lose your RAG !

Retrieval-augmented generation (RAG) in simplest terms, involves combining the input prompt for an LLM, with additional custom data and knowledge. An evolution of prompt engineering, it seeks and adds about data from outside of the LLMs original training datasets (hence augmented).

The simplified ‘RAG Lookup’ stage leverages a search optimized vector database, effectively ‘indexing’ an organisation’s local knowledge base and documents for use with LLMs. The LLM therefore can be given a factual grounding of the internal domain it previously wasn’t party to, regardless of the quantity of unstructured documents or number of legacy systems involved.

According to a 2023 report, 36.2% of Enterprise AI use cases employed RAG technology. An underappreciated fact is that to explore RAG and Prompt Engineering, data science expertise isn’t required. Whilst most organisations don’t have a wealth of data science talent, far more have DevOps engineers or application developers with Python or JavaScript programming skills. These are the AI Engineers of tomorrow.?

RAG Use Cases

Given foundational LLM’s generalist capabilities, the route to business value involves uncovering well defined use cases within each organisation. The ability to incorporate private company knowledge bases and specialist data feeds with GenAI opens up endless opportunities. Here a few broad examples to get you thinking;

·???????? Finance: Leveraging near real-time market data to optimise insight

·???????? Government: Conversational chatbots to help customers navigate complicated process

·???????? HR: Self-service conversational chatbots to improve employee onboarding

·???????? Insurance: Using the very latest internal and external data to reduce product risk

·???????? Legal: Combining the latest precedents and regulations to maximise compliance

·???????? Media: Fact checking journalists with additional news feeds to reduce litigation risk

·???????? Retail: Personalised e-commerce recommendations to improve loyalty

RAG Implication: RAG opens up organisation’s internal data and documents for exploration with GenAI’s semantic capability. Therefore generating GenAI business value is within reach (time, cost & skill) for many SMB and Enterprise businesses.

AI Agents

AI Agents, simply put, are autonomous software entities for use by humans, systems or indeed other agents. They may leverage multiple LLMs and can sense and act in their environment. AI Agents can perceive, think and react by themselves; they have agency.

AI Agents are generally goal-driven and break the overall task into a plan of smaller tasks using chain of thought. As well as generating their own plan, they can utilize other tools (such as web search and local RAG lookup), reflect and optimise, before executing the necessary action(s). Agents may also have a persona, for instance a business analyst, project manager or chat moderator.

By leveraging vector databases (covered in Part.1), AI Agents effectively obtain a memory. They therefore gain an understanding of their environment and previous interactions with it. Combining a memory with self-reflection gives AI Agents the ability to learn and adapt, to solve problems with ever-increasing efficiency. There are multiple types of agents, ranging from simple or ‘narrow agents’ that automate specific tasks, to more complex agents capable of collaborating other agents.

AI Agent Characteristics

·???????? Autonomy: Research, draft and revise their own actions to achieve their goal

·???????? Collaborative: With humans, multiple LLMs or other AI Agents

·???????? Memory: Dynamic short & long-term memory enables adaptability

·???????? Tool-aware: Ability to leverage tools enhances overall capability

AI Agent Implication: Previously, automation was a rule-based and static process. AI Agent’s ability to learn and adapt takes the prospect of self-enhancing automation to a whole new level.

Agentic Workflows

Agentic workflow is the multi-layered collaboration between multiple AI Agents within a GenAI Network (GAIN). Agents are assigned different roles and functions to best address complex problems and workflows may take over an hour to complete. Agentic workflow builds on prompt engineering techniques (but without humans) and RAG becomes agentic RAG, which can dynamically evaluate context and decide its own information retrieval path. Agentic workflow takes autonomous to a higher level and can combine with more traditional AI, such as Machine Learning. Rest easy, I’m not going to attempt to represent this visually !

Agentic Workflow Use Cases

Agentic workflow is rapidly evolving and represents the very forefront of AI use case exploration, from uncovering new analytic insight, driving automation or tackling complex reasoning and problem-solving.

·???????? E-Learning: Adapting to individual learner’s needs to deliver personalised learning and study guides

·???????? Energy: Smart grid controllers to optimise available resources based on demand forecasts and real-time energy prices

·???????? Healthcare: Combining the latest clinical data & research with medical knowledge to improve diagnosis for medical professionals

·???????? Legal: Combining jurisdiction-specific legal documentation with the latest precedent and regulatory compliance, to ease case research & preparation

·???????? SaaS Product: Next-gen product personalization and self-enhancing automation of Operations.

·???????? Scientific Research: Leveraging literature, research findings and the latest datasets to synthesise new insight and hypotheses

Agentic Workflow Implication: Realistically, complex agentic workflow will be out of reach for many Enterprise businesses from an agility, infrastructure and skills perspective. However, for product-based SaaS businesses especially, opportunities for next-level product features and operational automation seem boundless.

Conclusion: Generative AI is a potential game-changer to business, but with a notable caveat: it needs access to real-world context and supplementary specialist information to generate real business value. There’s a whole lot more to GenAI than being a productivity tool for individual users. In Part.3, we'll tackle where to start with an AI Readiness & AI Strategy.