Generative AI Deep Dive. (AI Agents, Multi-modality, RAG, Fine Tuning, Prompt Engineering)

By Kamal Atreja, Head of Delivery Ubique Digital LTD

Industry leaders remain optimistic about the transformative potential of Generative AI and its widespread adoption in the coming decade. Reports estimate that AI could contribute $20–$30 trillion to the global economy by 2030. Adoption rates are accelerating rapidly, as seen with OpenAI's ChatGPT, which has doubled API users since the mini-launch of ChatGPT-4. Generative AI and AI is reshaping industries like customer care, surveys, predictive analysis, translation, accounting, autonomous vehicles, and healthcare, fundamentally altering how AI integrates into our daily lives.

If you're looking to understand the basics of Generative AI and its significance, please explore further on my previous post – LinkedIn- Generative AI Basics????

As we now are aware of Generative AI and that the power to generate content like stories, images, video, music, art is not just limited to humans, let’s take proceed to understand a little deep into Generative AI and covers some areas like AI Agents, Multi-modality, RAG, Model Tuning and Prompt Engineering. Each of them in itself is a domain to understand and work with but how do all these come together towards generating a human-like response. This is about going from monolithic models to compound AI models.

AI Agents?

AI Agents (Artificial Intelligence Agents) can be thought of as skilled functions designed to leverage AI systems (often with vast capabilities) to accomplish specific tasks. Here’s how they work in a simplified sequence and then Iterate to create the most relevant results expected.

1. Access Memory: AI agents can understand the context of the people or processes they interact with by accessing memory, including prior knowledge of completed tasks.
2. Reason: They possess the ability to think critically and evaluate the task at hand.
3. Act: Based on their reasoning, they take necessary actions to achieve the desired outcome.
4. Iterative Re-ACT: If the result doesn’t meet expectations, AI agents can revisit their memory of prompts, reason again, and iteratively act to refine their output until the results are relevant and accurate.

This iterative process enables AI agents to adapt dynamically, ensuring precision and effectiveness in achieving goals.

Multi-Modality In the world of AI models, key concepts include Uni-Modal, Multi-Modal, and Cross-Modal systems:

1. Uni-Modal Models: These handle a single type of input and produce a corresponding output. For example, ChatGPT-3 processes text as input and generates text as output.
2. Multi-Modal Models: These can process multiple types of inputs—such as images, audio, text, or video—and generate outputs in multiple forms. ChatGPT-4 is an example, as it can take inputs like text, images, or audio and provide outputs in text, images, or audio. Behind the scenes, systems like Whisper (for audio-to-text) and DALL·E (for text-to-image) work in tandem to deliver results. Multi-Modal AI is particularly powerful because of its versatility, such as Google's Gemini, which handles any input type and generates outputs across different modes. For instance, a Multi-Modal AI in healthcare could synthesize patient data from text, medical reports, images, and audio to provide accurate diagnoses.
3. Cross-Modal Models: These take a single input type and produce a different type of output. For instance, OpenAI’s AORA converts text to videos, while Google’s MUSICFX translates text into music. These systems excel at transforming one modality into another.

Multi-Modal AI is often seen as the most comprehensive, as it handles diverse inputs and outputs seamlessly, enabling use cases across industries. For example, in a clinical scenario, it can integrate diagnostic reports, verbal interactions, motion analysis, and medical imagery to provide well-rounded recommendations.

We’ll leave deeper discussions of Multi-Modal systems here and transition to the topic of Retrieval-Augmented Generation (RAG) next.

Retrieval-Augmented Generation (RAG)

RAG leverages the capabilities of Large Language Models (LLMs) or other foundational models while incorporating custom content and context for more precise and relevant outputs.

How RAG Works

Consider a chatbot designed to answer HR-related queries from employees. For example, if an employee asks about their sick leave policy, the chatbot typically sends a prompt to the LLM, which responds with a generic answer based on publicly available internet data. However, with RAG:

1. Retrieval: The system retrieves specific organizational data, such as the employee's sick leave history and company sick leave policies.
2. Augmentation: This retrieved data is combined with the original prompt and instructions, ensuring the LLM receives contextualized, relevant input.
3. Generation: The LLM uses this augmented input to generate a highly tailored response, offering much more value to the employee than a generic answer.

The Role of Vectors and Vector Databases

To enhance RAG further, the applications use vector databases.?

• All policy documents, employee records, and other relevant content are converted into numerical vectors.
• In vector space, similar topics are represented by equally distanced vectors, enabling quick retrieval of contextually relevant data.
• When an employee asks a question, RAG retrieves the relevant vectorized content (e.g., policy excerpts, personal records) based on similarity and provides it to the LLM as embeddings along with the original query.

By combining these elements, RAG ensures responses are precise, actionable, and tailored to the user’s needs. In upcoming discussions, we will dive deeper into setting up RAG architectures, using vector databases effectively, and best practices for integrating RAG into enterprise systems.

Fine Tuning Fine-tuning is a process used to adapt Large Language Models (LLMs) to perform well in specialized domains such as legal documents, domain-specific datasets, or industry-targeted responses. This ensures the model generates accurate, contextual, and specific responses rather than generic or irrelevant ones. Fine-tuning involves training a pre-existing base model on a domain-specific dataset to align its responses with specialized requirements.

Key Benefits of Fine-Tuning:

1. Contextual Precision: Fine-tuning allows LLMs to deliver responses tailored to a specific domain or product.
2. Specialized Knowledge: Models adapt to the nuances of a specialized field, such as legal, medical, or financial sectors, ensuring accurate answers.
3. Reduced Hallucination: Fine-tuned models are less likely to generate unrelated or misleading responses since they learn from focused datasets.

The fine-tuning process begins with selecting a general-purpose LLM that has been pre-trained on broad datasets. This base model is then refined using domain-specific data, such as legal cases or financial records, to adapt its parameters to the specific needs of the target domain. Once the model is trained, it undergoes validation, where it is tested on curated datasets to assess its accuracy and contextual relevance. Finally, its performance is evaluated, and parameters are iteratively adjusted to optimize the results, ensuring the model delivers precise and specialized outputs.

RAG vs. Fine-Tuning:

Retrieval-Augmented Generation (RAG) and Fine-Tuning both enhance the relevance of LLM responses, but they take distinct approaches. Fine-Tuning is well-suited for static, domain-specific knowledge, deeply adapting the model to specialized datasets; however, it is limited to the knowledge available up to its training cut-off date. On the other hand, RAG excels in dynamic scenarios where real-time or continuously updated information, such as current events, needs to supplement the model's outputs.?

A combined approach often proves effective. For instance:

• Fine-tuning handles static domain expertise.
• RAG provides dynamic, up-to-date contextual information.

Methods of Fine-Tuning:

1. Self-Supervised Learning: The model learns from vast unlabeled datasets without human intervention.
2. Supervised Learning: Humans curate and label datasets, guiding the model toward specific output targets.
3. Reinforcement Learning: Models iteratively improve by ranking responses, selecting the best, and refining future outputs (e.g., RLHF – Reinforcement Learning with Human Feedback).

This brings us to the final topic of today “Prompt Engineering”

Prompt Engineering Prompt Engineering is the art and science of effectively communicating with Large Language Models (LLMs) to elicit the most relevant and accurate responses. LLMs, trained on vast datasets from the internet, are incredibly versatile, capable of providing insights on topics ranging from healthcare and legal documents to sports. However, the quality of their output depends significantly on how the prompts are structured and presented.

At its core, Prompt Engineering involves designing and creating well-crafted questions or instructions to guide LLMs in producing contextually accurate and meaningful responses. For example, while a pre-trained LLM can generate detailed outputs, its responses may occasionally suffer from biases, hallucinations (generating incorrect or fabricated information), or misinterpretations due to limitations in its training data or ambiguous prompts.

To ensure quality and relevance, it is crucial to critically evaluate the output of LLMs and iteratively refine the prompts. Factors like the phrasing, structure, and specificity of a prompt can significantly influence the model's performance. For instance, asking a model for travel recommendations or document summarization requires precise and context-rich prompts to yield useful results.

As LLMs become increasingly integrated into workflows, the importance of Prompt Engineering grows. It bridges the gap between human intent and machine understanding, ensuring outputs align with user expectations. This evolving skill set highlights the need for prompt engineers who can refine and optimize interactions with AI systems.

Please contact us if you would like to chat more about AI for your organisation.

Until next time,

Kamal Atreja