Unlocking Success: Why Gen AI Product Managers Must Master 'The Treacherous Twelve' pitfalls in RAG Architectures.

Retrieval Augmented Generation is the talk of the town for LLM Application Architects; touting its capability to improve the quality of outputs from large language models (LLMs) used in Generative AI applications. It is supposed to reduce the chance of false or inaccurate outputs — what are usually called ‘hallucinations’. It works by augmenting outputs with information from a verified source. A Retrieval Augmented Generation (RAG) system combines information retrieval capabilities with the generative prowess of LLMs.

However, AI Product Managers and Gen AI Application Architects must be cognizant of 12 points of possible failure in a RAG Architecture. Understanding these failure points is crucial for designing robust and reliable RAG systems. Addressing these challenges requires careful consideration of content curation, retrieval algorithms, context consolidation strategies, language model fine-tuning, and continuous monitoring and improvement of the system.

By proactively addressing these failure points and designing architectures that allow for continuous improvement, AI Product Managers and AI Applications Architects can create RAG systems that deliver accurate, relevant, and trustworthy information to users, enhancing the overall value and impact of the AI application.

Failure Point 1: Missing Content

Failure Point 2: Missed the Top Ranked Documents

Failure Point 3: Not in Context — Consolidation Strategy Limitations

Failure Point 4: Not Extracted

Failure Point 5: Wrong Format

Failure Point 6: Incorrect Specificity

Failure Point 7: Incomplete

Failure Point 8: Data Ingestion Scalability

Failure Point 9: Structured Data QA

Failure Point 10: Data Extraction from Complex PDFs

Failure Point 11: Fallback Model(s)

Failure Point 12: LLM Security

Before we deep dive into these failure points and possible solution / mitigation approaches, lets first discuss the drive to implementing RAG and the value it adds.

RAG is being used in a number of ways. It’s being used in search engines, for example, to better understand search intent and provide more relevant results. RAG is also improving the quality of chatbots, helping them to deliver more accurate results, thus providing a much better customer experience and, ultimately, ensuring generative AI is more impactful for a business.

It’s also particularly useful in contexts like translation and document summarization — using a RAG architecture that retrieves knowledge from specific resources can ensure that the system's outputs are more reliable.

What are some problems solved by RAG?

Information Overload:

Issue: Large language models can struggle to manage vast amounts of data effectively.

Solution: RAG selectively retrieves the most relevant information, ensuring the model focuses on pertinent data.

Static Knowledge Limitation:

Issue: LLMs have a fixed knowledge base that can become outdated.

Solution: RAG integrates dynamic retrieval mechanisms to access the latest information.

Factual Accuracy:

Issue: LLMs can generate inaccurate or misleading information.

Solution: RAG enhances factual accuracy by sourcing verified information during the generation process.

Contextual Understanding:

Issue: LLMs may provide generic or irrelevant responses without adequate context.

Solution: RAG improves contextual understanding by incorporating contextually relevant data into the generation process.

What is the value delivered by RAG Architecture into a LLM application?

Improved Accuracy:

Value: Enhances the reliability of responses by grounding them in retrieved factual information.

Contextual Relevance:

Value: Produces more relevant and context-aware answers, improving user satisfaction and trust.

Dynamic Updating:

Value: Keeps the generated content current by continuously integrating the latest information from external sources.

Enhanced Personalization:

Value: Delivers more personalized responses by leveraging user-specific data, leading to better user experiences.

Resource Efficiency:

Value: Reduces the need for extensive model training on vast datasets, leveraging retrieval mechanisms to access necessary information efficiently.

What are the 12 failure points?

1. Missing Content: The RAG system fails to provide an answer because the necessary information is not available in the knowledge base. This is significant because it highlights the importance of having comprehensive and up-to-date content in the retrieval system.
2. Missed Top Ranked Documents: The answer exists in the knowledge base, but the relevant documents did not rank high enough to be returned. This failure point emphasizes the need for effective ranking algorithms to surface the most pertinent information.
3. Consolidation Strategy Limitations: The relevant documents are retrieved, but the information is not effectively consolidated into the context passed to the language model. This underscores the importance of developing robust strategies for combining and summarizing retrieved content.
4. Incorrect Answer Extraction: The correct answer is present in the context, but the language model fails to extract it accurately. This points to potential limitations in the language model's ability to understand and extract specific information from the provided context.
5. Wrong Format: The generated answer does not adhere to the desired format, such as a table or list. This highlights the need for clear formatting instructions and the language model's capability to follow them.
6. Incorrect Specificity: The answer lacks the appropriate level of detail or is too specific for the given question. This emphasizes the importance of fine-tuning the system to provide answers at the right level of granularity for the target audience and use case.
7. Incomplete Answers: The generated response is not incorrect but lacks some of the necessary information, even though it was available in the knowledge base. This underscores the need for the language model to effectively utilize all relevant information from the retrieved context.
8. Data Ingestion Scalability: The data ingestion pipeline faces challenges in scaling up for higher data volumes, especially in research and biomedical domains. This highlights the importance of optimizing data ingestion processes and infrastructure to handle large-scale datasets.
9. Structured Data QA: Interpreting user requests to extract relevant structured data accurately poses challenges, particularly with complex or unclear queries. This emphasizes the need for robust query understanding and information extraction techniques.
10. Data Extraction from Complex PDFs: Extracting data from various types of documents, such as PDFs with embedded tables or images, can be challenging. This underscores the importance of developing specialized techniques for parsing and extracting information from complex document formats.
11. Fallback Model(s): The need for backup models in case of primary model malfunctions, particularly important in high-stakes biomedical applications. This highlights the significance of designing failsafe mechanisms and redundancy in RAG systems.
12. LLM Security: Addressing prompt injection and insecure outputs, critical for sensitive educational and biomedical data. This emphasizes the importance of implementing security measures to prevent unauthorized access and misuse of the system.

Here's an example of a LLM application using Retrieval Augmented Generation (RAG) and how the twelve key failure points can impact its scalability and performance,

Example Application: A Customer Support Chatbot for a software company that uses RAG to answer user queries by retrieving relevant information from the product documentation, knowledge base articles, and FAQ pages.

Potential Impacts of RAG Failure Points:

1. Missing Content: If the Chatbot's knowledge base is missing crucial information about a product feature, it may fail to provide a satisfactory answer to a user's question, leading to a poor user experience and increased escalations to human support.
2. Missed Top Ranked Documents: If the Retrieval system fails to surface the most relevant documents for a query, the Chatbot may provide irrelevant or incomplete answers, frustrating users and eroding trust in the application.
3. Consolidation Strategy Limitations: If the Chatbot fails to effectively combine information from multiple sources into a coherent response, users may receive disjointed or contradictory answers, diminishing the application's perceived intelligence and reliability.
4. Incorrect Answer Extraction: If the language model misinterprets the context and extracts the wrong information, the chatbot may provide inaccurate answers, leading to misinformed users and potential business risks.
5. Wrong Format: If the chatbot generates responses in an unexpected or inconsistent format, such as providing a code snippet when asked for a high-level explanation, users may find the answers difficult to understand or apply, reducing the application's usefulness.
6. Incorrect Specificity: If the chatbot provides overly technical answers to novice users or oversimplified responses to experts, it may fail to meet the needs of different user segments, limiting adoption and satisfaction.
7. Incomplete Answers: If the chatbot omits important details or steps in its responses, users may struggle to resolve their issues independently, leading to increased support costs and reduced efficiency gains from the application.
8. Data Ingestion Scalability: As the software company's documentation grows, the chatbot may face challenges in efficiently processing and indexing new content, resulting in outdated or incomplete knowledge that affects answer quality.
9. Structured Data QA: If the chatbot struggles to interpret queries involving specific product versions, operating systems, or configurations, it may provide generic or inapplicable answers, diminishing its value for troubleshooting complex issues.
10. Data Extraction from Complex PDFs: If the chatbot fails to accurately extract information from PDFs with intricate layouts or embedded images, it may miss critical details needed to answer user questions comprehensively.
11. Fallback Model(s): Without robust fallback mechanisms, the chatbot may simply fail to respond when the primary RAG system encounters errors, leaving users without assistance and damaging the application's reliability.
12. LLM Security: If the chatbot is vulnerable to prompt injection attacks or leaks sensitive information in its responses, it could expose the company to data breaches, intellectual property theft, or reputational damage

Reference [2] lists all 12 RAG failure points and their proposed solutions side by side in a table. Please refer to [2[ for detailed explanation of the solution approaches,

Applying these 12 solution approaches to the Customer Chatbot example one can tackle the potential 12 failure points in a graded manner.

1. Missing Content

Pain Point: Context missing in the knowledge base.

Proposed Solution: Clean your data & Better prompting.

Impact: Ensuring the knowledge base is comprehensive and up-to-date will reduce instances of missing content, leading to more accurate and complete responses. Better prompting can help the language model generate more relevant queries to retrieve the necessary information.

2. Missed Top Ranked Documents

Pain Point: Context missing in the initial retrieval pass.

Proposed Solution: Hyperparameter tuning & Reranking.

Impact: Fine-tuning the retrieval model's hyperparameters and implementing reranking strategies will improve the relevance of the top-ranked documents, ensuring that the most pertinent information is surfaced for the chatbot to use.

3. Not in Context - Consolidation Strategy Limitations

Pain Point: Context missing after reranking.

Proposed Solution: Tweak retrieval strategies & Finetune embeddings.

Impact: Adjusting retrieval strategies and fine-tuning embeddings will enhance the system's ability to consolidate relevant information into the context, leading to more coherent and accurate responses from the chatbot.

4. Not Extracted

Pain Point: Context not extracted.

Proposed Solution: Clean your data, prompt compression, & LongContextReorder.

Impact: Cleaning data and using prompt compression techniques will help the language model better understand and extract the necessary information. LongContextReorder can help manage longer contexts, ensuring that critical information is not missed.

5. Wrong Format

Pain Point: Output is in the wrong format.

Proposed Solution: Better prompting, output parsing, pydantic programs, & OpenAI JSON mode.

Impact: Improved prompting and output parsing will ensure that the chatbot's responses are in the correct format, making them easier for users to understand and apply. Pydantic programs and JSON mode can help structure the output consistently.

6. Incorrect Specificity

Pain Point: Output has an incorrect level of specificity.

Proposed Solution: Advanced retrieval strategies.

Impact: Implementing advanced retrieval strategies will help tailor the specificity of the responses to match the user's needs, whether they require high-level overviews or detailed technical explanations.

7. Incomplete

Pain Point: Output is incomplete.

Proposed Solution: Query transformations.

Impact: Using query transformations can help the chatbot generate more comprehensive responses by ensuring that all relevant aspects of a query are addressed, reducing the likelihood of incomplete answers.

8. Data Ingestion Scalability

Pain Point: Ingestion pipeline can't scale to larger data volumes.

Proposed Solution: Parallelizing ingestion pipeline.

Impact: Parallelizing the data ingestion pipeline will improve the system's ability to handle large volumes of data efficiently, ensuring that the knowledge base remains up-to-date and comprehensive.

9. Structured Data QA

Pain Point: Inability to QA structured data.

Proposed Solution: Chain-of-table pack & Mix-self-consistency pack.

Impact: These techniques will enhance the system's ability to accurately retrieve and QA structured data, ensuring that the chatbot can provide precise and reliable answers based on structured information.

10. Data Extraction from Complex PDFs

Pain Point: Document (PDF) parsing.

Proposed Solution: Embedded table retrieval.

Impact: Implementing embedded table retrieval will improve the chatbot's ability to extract and utilize information from complex PDFs, ensuring that critical details are not overlooked.

11. Fallback Model(s)

Pain Point: Rate limit errors.

Proposed Solution: Neutrino router & OpenRouter.

Impact: Using fallback models like Neutrino router and OpenRouter will ensure that the chatbot remains functional even when the primary model encounters errors or rate limits, enhancing the system's reliability and user experience.

12. LLM Security

Pain Point: Prompt injection, etc.

Proposed Solution: NeMo Guardrails & Llama Guard.

Impact: Implementing security measures like NeMo Guardrails and Llama Guard will protect the chatbot from prompt injection attacks and other security vulnerabilities, ensuring the integrity and safety of the system.

All of these proposed solutions are not mutually exclusive and one has to consider trade-offs between them. I shall cover these trade-offs in a follow up article as more research and study is needed.

In summary, LLM applications have four distinct patterns in their architectural evolution as the nature of the issues, complexity of the application, compute and performance and scalability requirements changes. RAG seems to have come as a landing point for LLM Architectural evolution. Prompt Engineering is a first step, Fine Tuning has not shown to provide drastic improvements in practice and Pre-Training needs building models from scratch which maybe beyond the need of many organizations.

Overall, this means that RAG architectures must be audited thoroughly for robustness before putting them into production, and you can easily shoot yourself in the foot by releasing an agent or a chatbot that hasn’t gone through considerations for mitigating the Treacherous Twelve failure points.

References:

1. Seven Failure Points When Engineering a Retrieval Augmented Generation System. https://arxiv.org/abs/2401.05856
2. RAG Pain Points and Proposed Solutions - Solving the core challenges of Retrieval-Augmented Generation. https://towardsdatascience.com/12-rag-pain-points-and-proposed-solutions-43709939a28c