Enhancing RAG-Based Solutions with Intelligent Context Retrieval

Introduction

As we progress into an era dominated by Artificial Intelligence (AI) and machine learning, tools like Langchain, Large Language Models (LLMs), and generative AI are at the forefront of transforming various industries. These technologies promise to revolutionize how we interact with data, automate processes, and enhance decision-making capabilities. Their significance lies in the ability to provide intelligent, contextually aware responses that streamline operations and improve user experiences.

RAG-Based Architecture

Retrieval-Augmented Generation (RAG) architecture enhances the relevance and quality of responses by integrating retrieval and generation mechanisms. Here's how RAG works:

Components:

• Retriever Module: Upon receiving a query, the retriever module searches through stored embeddings to fetch relevant passages from the database.
• Generator Module: These retrieved passages are then fed into a language model (LLM) such as GPT, which combines the input query with the retrieved context to generate a coherent and informative response.

Retrieving Contexts:

When a query is made, the retriever module identifies several relevant contexts from the database. The number of these contexts is often denoted by K. The choice of K is crucial, as it determines the amount of information the LLM will consider when generating a response.

Metrics for Evaluating LLM Responses

To assess the quality of responses generated by LLMs, we use the following metrics:

• Faithfulness: Ensures all claims in the generated answer can be inferred from the context.
• Answer Relevancy: Measures how well the generated answer addresses the given prompt.
• Context Recall: Evaluates how well the retrieval context aligns with the ground truth.
• Context Precision: Checks if all relevant items are ranked higher in the context.
• Context Relevancy: Assesses the relevancy of the retrieved context to the query.

The Role of K in Context Retrieval

K represents the number of contexts retrieved by the retriever module. While providing more contexts (a higher K) may seem beneficial, it also poses challenges:

• Cost: The more contexts sent to the LLM, the more tokens it utilizes, leading to higher computational and financial costs.
• Relevance: If information is not well-distributed across contexts, including too many contexts can result in additional irrelevant or incorrect add-ons to the already correct answer.
• Faithfulness: Ensuring all claims in the generated answer can be inferred from the context is crucial. Including too many contexts can dilute the relevance, reducing faithfulness.

The Challenge

In practice, the chatbot faced issues where LLMs produced answers containing irrelevant information. LLMs tend to rank the initial chunks higher than the subsequent ones, meaning the generated answer predominantly comes from the initial chunks (chunks with higher similarity). However, this does not guarantee that the LLM will not output unnecessary information from the remaining chunks. This often occurred because the essential information was confined to a single context, while LLMs tended to include parts from multiple contexts. This led to decreased answer relevancy, correctness, and faithfulness.

Optimizing K-Contexts

To address this, we focused on optimizing the value of K by leveraging the retrieval function's scoring system:

• Retrieval Function: similarity_search_with_relevance_scores provides similarity scores for each context based on the query.
• Threshold Setting: We set a threshold to determine which contexts had sufficiently high similarity scores. Contexts meeting or exceeding this threshold were considered relevant.
• Filter Contexts: By comparing the similarity and relevance scores among the retrieved contexts, we apply threshold filtering, which results in dynamic filtering of the contexts which do not surpass the threshold criteria.

The threshold values depend on the data's distribution. If the data is diverse and spread across different chunks, the scores will be closer. A greater difference in scores indicates less similarity.

Here is a code snippet for determining the optimal value of K

def determine_k_for_question(question, threshold):
    """
    Determines the optimal value of K for a given question based on similarity scores.

    Args:
    question (str): The input question.
    threshold (float): The threshold for considering a similarity score as sufficiently high.

    Returns:
    int: The selected value of K.
    """
    # Retrieve similarity scores for the question from the database
    scores = db.similarity_search_with_relevance_scores(question)
    
    # Initialize the value of K
    k = 0

    # Iterate through the scores to determine the value of K
    for i, score in enumerate(scores):
        if (i == 0):
            # Increment K for the first score
            k += 1
            # Set a benchmark score for comparison
            bench_score = score[1] - threshold
        elif (score[1] > bench_score):
            # Increment K if the current score exceeds the benchmark score
            k += 1
    
    return k
        

Results

Implementing this approach led to significant improvements in the chatbot's performance:

• Faithfulness: Increased by 31.41%
• Answer Relevancy: Increased by 5.8%
• Rest of the evaluation metrics remained comparable.
• Efficiency: Reduced the number of tokens processed, lowering computational and financial costs.
• Precision: Enhanced the accuracy of responses by focusing on the most pertinent information.

Conclusion

By optimizing the value of K in RAG-based architecture, we significantly improved the throughput and faithfulness of our AI-enabled chatbot. The higher faithfulness and answer relevancy observed indicates that the generated responses align more closely with the information present in the retrieved contexts. This intelligent approach not only reduced computational costs but also ensured higher quality and more reliable responses, showcasing the transformative potential of combining retrieval mechanisms with generative AI in developing intelligent conversational agents.