Advanced RAG Techniques Part 2: Querying and Testing

Welcome to Part 2 of our article on Advanced RAG Techniques! In part 1 of this series, we set-up, discussed, and implemented the data processing components of the advanced RAG pipeline:

In this part, we're going to proceed with querying and testing out our implementation. Let's get right to it!

Table of Contents

1. Searching and Retrieving, Generating Answers
2. HyDE (Hypothetical Document Embedding)
3. Hybrid Search
4. Test 1: Who audits Elastic?
5. Test 2: What was the total revenue in 2023?
6. Test 3: What product does growth primarily depend on? How much?
7. Test 4: Describe employee benefit plan
8. Test 5: Which companies did Elastic acquire?
9. Conclusion

Searching and Retrieving, Generating Answers

Let's ask our first query, ideally some piece of information found primarily in the annual report. How about:

Who audits Elastic?        

Now, let's apply a few of our techniques to enhance the query.

Enriching Queries with Synonyms

Firstly, let's enhance the diversity of the query wording, and turn it into a form that can be easily processed into an Elasticsearch query. We'll enlist the aid of GPT-4o to convert the query into a list of OR clauses. Let's write this prompt:

ELASTIC_SEARCH_QUERY_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating Elasticsearch query strings. Your task is to create the most effective query string for the given user question. This query string will be used to search for relevant documents in an Elasticsearch index.

Guidelines:
1. Analyze the user's question carefully.
2. Generate ONLY a query string suitable for Elasticsearch's match query.
3. Focus on key terms and concepts from the question.
4. Include synonyms or related terms that might be in relevant documents.
5. Use simple Elasticsearch query string syntax if helpful (e.g., OR, AND).
6. Do not use advanced Elasticsearch features or syntax.
7. Do not include any explanations, comments, or additional text.
8. Provide only the query string, nothing else.

For the question "What is Clickthrough Data?", we would expect a response like:
clickthrough data OR click-through data OR click through rate OR CTR OR user clicks OR ad clicks OR search engine results OR web analytics

AND operator is not allowed. Use only OR.

User Question:
[The user's question will be inserted here]

Generate the Elasticsearch query string:
'''        

When applied to our query, GPT-4o generates synonyms of the base query and related vocabulary.

'audits elastic OR 
elasticsearch audits OR 
elastic auditor OR 
elasticsearch auditor OR 
elastic audit firm OR 
elastic audit company OR 
elastic audit organization OR 
elastic audit service'        

In the ESQueryMaker class, I've defined a function to split the query:

def parse_or_query(self, query_text: str) -> List[str]:
    # Split the query by 'OR' and strip whitespace from each term
    # This converts a string like "term1 OR term2 OR term3" into a list ["term1", "term2", "term3"]
    return [term.strip() for term in query_text.split(' OR ')]        

Its role is to take this string of OR clauses and split them into a list of terms, allowing us do a multi-match on our key document fields:

["original_text", 'keyphrases', 'potential_questions', 'entities']        

Finally ending up with this query:

 'query': {
    'bool': {
        'must': [
            {
                'multi_match': {
                'query': 'audits Elastic Elastic auditing Elastic audit process Elastic compliance Elastic security audit Elasticsearch auditing Elasticsearch compliance Elasticsearch security audit',
                'fields': [
                    'original_text',
                'keyphrases',
                'potential_questions',
                'entities'
                ],
                'type': 'best_fields',
                'operator': 'or'
                }
            }
      ]        

This covers many more bases than the original query, hopefully reducing the risk of missing a search result because we forgot a synonym. But we can do more.

HyDE (Hypothetical Document Embedding)

Let's enlist GPT-4o again, this time to implement HyDE.

The basic premise of HyDE is to generate a hypothetical document - The kind of document that would likely contain the answer to the original query.

The factuality or accuracy of the document is not a concern. With that in mind, let's write the following prompt:

HYDE_DOCUMENT_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating hypothetical documents based on user queries. Your task is to create a detailed, factual document that would likely contain the answer to the user's question. This hypothetical document will be used to enhance the retrieval process in a Retrieval-Augmented Generation (RAG) system.

Guidelines:
1. Carefully analyze the user's query to understand the topic and the type of information being sought.
2. Generate a hypothetical document that:
   a. Is directly relevant to the query
   b. Contains factual information that would answer the query
   c. Includes additional context and related information
   d. Uses a formal, informative tone similar to an encyclopedia or textbook entry
3. Structure the document with clear paragraphs, covering different aspects of the topic.
4. Include specific details, examples, or data points that would be relevant to the query.
5. Aim for a document length of 200-300 words.
6. Do not use citations or references, as this is a hypothetical document.
7. Avoid using phrases like "In this document" or "This text discusses" - write as if it's a real, standalone document.
8. Do not mention or refer to the original query in the generated document.
9. Ensure the content is factual and objective, avoiding opinions or speculative information.
10. Output only the generated document, without any additional explanations or meta-text.

User Question:
[The user's question will be inserted here]

Generate a hypothetical document that would likely contain the answer to this query:
'''        

Since vector search typically operates on cosine vector similarity, the premise of HyDE is that we can achieve better results by matching documents to documents instead of queries to documents.

What we care about is structure, flow, and terminology. Not so much factuality. GPT-4o outputs a HyDE document like this:

'Elastic N.V., the parent company of Elastic, the organization known for developing Elasticsearch, is subject to audits to ensure financial accuracy, 
regulatory compliance, and the integrity of its financial statements. The auditing of Elastic N.V. is typically conducted by an external, 
independent auditing firm. This is common practice for publicly traded companies to provide stakeholders with assurance regarding the company\'s 
financial position and operations....        

It looks pretty believable, like the ideal candidate for the kinds of documents we'd like to index. We're going to embed this and use it for hybrid search.

Hybrid Search

This is the core of our search logic. Our lexical search component will be the generated OR clause strings.

Our dense vector component will be embedded HyDE Document (aka the search vector). We use KNN to efficiently identify several

candidate documents closest to our search vector. We call our lexical search component Scoring with TF-IDF and BM25 by default.

Finally, the lexical and dense vector scores will be combined using the 30/70 ratio recommended by Wang et al.

def hybrid_vector_search(self, index_name: str, query_text: str, query_vector: List[float], 
                         text_fields: List[str], vector_field: str, 
                         num_candidates: int = 100, num_results: int = 10) -> Dict:
    """
    Perform a hybrid search combining text-based and vector-based similarity.

    Args:
        index_name (str): The name of the Elasticsearch index to search.
        query_text (str): The text query string, which may contain 'OR' separated terms.
        query_vector (List[float]): The query vector for semantic similarity search.
        text_fields (List[str]): List of text fields to search in the index.
        vector_field (str): The name of the field containing document vectors.
        num_candidates (int): Number of candidates to consider in the initial KNN search.
        num_results (int): Number of final results to return.

    Returns:
        Dict: A tuple containing the Elasticsearch response and the search body used.
    """
    try:
        # Parse the query_text into a list of individual search terms
        # This splits terms separated by 'OR' and removes any leading/trailing whitespace
        query_terms = self.parse_or_query(query_text)

        # Construct the search body for Elasticsearch
        search_body = {
            # KNN search component for vector similarity
            "knn": {
                "field": vector_field,  # The field containing document vectors
                "query_vector": query_vector,  # The query vector to compare against
                "k": num_candidates,  # Number of nearest neighbors to retrieve
                "num_candidates": num_candidates  # Number of candidates to consider in the KNN search
            },
            "query": {
                "bool": {
                    # The 'must' clause ensures that matching documents must satisfy this condition
                    # Documents that don't match this clause are excluded from the results
                    "must": [
                        {
                            # Multi-match query to search across multiple text fields
                            "multi_match": {
                                "query": " ".join(query_terms),  # Join all query terms into a single space-separated string
                                "fields": text_fields,  # List of fields to search in
                                "type": "best_fields",  # Use the best matching field for scoring
                                "operator": "or"  # Match any of the terms (equivalent to the original OR query)
                            }
                        }
                    ],
                    # The 'should' clause boosts relevance but doesn't exclude documents
                    # It's used here to combine vector similarity with text relevance
                    "should": [
                        {
                            # Custom scoring using a script to combine vector and text scores
                            "script_score": {
                                "query": {"match_all": {}},  # Apply this scoring to all documents that matched the 'must' clause
                                "script": {
                                    # Script to combine vector similarity and text relevance
                                    "source": """
                                    # Calculate vector similarity (cosine similarity + 1)
                                    # Adding 1 ensures the score is always positive
                                    double vector_score = cosineSimilarity(params.query_vector, params.vector_field) + 1.0;
                                    # Get the text-based relevance score from the multi_match query
                                    double text_score = _score;
                                    # Combine scores: 70% vector similarity, 30% text relevance
                                    # This weighting can be adjusted based on the importance of semantic vs keyword matching
                                    return 0.7 * vector_score + 0.3 * text_score;
                                    """,
                                    # Parameters passed to the script
                                    "params": {
                                        "query_vector": query_vector,  # Query vector for similarity calculation
                                        "vector_field": vector_field  # Field containing document vectors
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }

        # Execute the search request against the Elasticsearch index
        response = self.conn.search(index=index_name, body=search_body, size=num_results)
        # Log the successful execution of the search for monitoring and debugging
        logger.info(f"Hybrid search executed on index: {index_name} with text query: {query_text}")
        # Return both the response and the search body (useful for debugging and result analysis)
        return response, search_body
    except Exception as e:
        # Log any errors that occur during the search process
        logger.error(f"Error executing hybrid search on index: {index_name}. Error: {e}")
        # Re-raise the exception for further handling in the calling code
        raise e        

Finally, we can piece together a RAG function. Our RAG, from query to answer, will follow this flow:

1. Convert Query to OR Clauses.

2. Generate HyDE document and embed it.

3. Pass both as inputs to Hybrid Search.

4. Retrieve top-n results, reverse them so that the most relevant score is the "most recent" in the LLM's contextual memory (Reverse Packing)

Reverse Packing Example:

Query: "Elasticsearch query optimization techniques"

Retrieved documents (ordered by relevance):

1. "Use bool queries to combine multiple search criteria efficiently."

2. "Implement caching strategies to improve query response times."

3. "Optimize index mappings for faster search performance."

Reversed order for LLM context:

3. "Optimize index mappings for faster search performance."

2. "Implement caching strategies to improve query response times."

1. "Use bool queries to combine multiple search criteria efficiently."

By reversing the order, the most relevant information (1) appears last in the context, potentially receiving more attention from the LLM during answer generation.

5. Pass the context to the LLM for generation.

def get_context(index_name, 
                match_query, 
                text_query, 
                fields, 
                num_candidates=100, 
                num_results=20, 
                text_fields=["original_text", 'keyphrases', 'potential_questions', 'entities'], 
                embedding_field="primary_embedding"):

    embedding=embedder.get_embeddings_from_text(text_query)

    results, search_body = es_query_maker.hybrid_vector_search(
        index_name=index_name,
        query_text=match_query,
        query_vector=embedding[0][0],
        text_fields=text_fields,
        vector_field=embedding_field,
        num_candidates=num_candidates,
        num_results=num_results
    )

    # Concatenates the text in each 'field' key of the search result objects into a single block of text.
    context_docs=['\n\n'.join([field+":\n\n"+j['_source'][field] for field in fields]) for j in results['hits']['hits']]

    # Reverse Packing to ensure that the highest ranking document is seen first by the LLM.
    context_docs.reverse()
    return context_docs, search_body

def retrieval_augmented_generation(query_text):
    match_query= gpt4o.generate_query(query_text)
    fields=['original_text']

    hyde_document=gpt4o.generate_HyDE(query_text)

    context, search_body=get_context(index_name, match_query, hyde_document, fields)

    answer= gpt4o.basic_qa(query=query_text, context=context)
    return answer, match_query, hyde_document, context, search_body        

Let's run our query and get back our answer:

According to the context, Elastic N.V. is audited by an independent registered public accounting firm, PricewaterhouseCoopers (PwC). 
This information is found in the section titled "report of independent registered public accounting firm," which states:

"We have audited the accompanying consolidated balance sheets of Elastic N.V. [...] / s / pricewaterhouseco."        

Nice. That's correct.

Experiments

There's an important question to answer now. What did we get out of investing so much effort and additional complexity into these implementations?

Let's do a little comparison. The RAG pipeline we've implemented versus baseline hybrid search, without any of the enhancements we've made.

We'll run a small series of tests and see if we notice any substantial differences. We'll refer to the RAG we have just implemented as AdvancedRAG, and the basic pipeline as SimpleRAG.

Test 1: Who audits Elastic?

AdvancedRAG

According to the context, Elastic N.V. is audited by an independent registered public accounting firm, PricewaterhouseCoopers (PwC).         

SimpleRAG

According to the provided context, no specific auditing firm or company is mentioned as the auditor for Elastic.         

Summary: SimpleRAG did not identify PWC as the auditor

Okay that's actually quite surprising. That looks like a search failure on SimpleRAG's part. No documents related to auditing were retrieved.

Let's dial down the difficulty a little with the next test.

Test 2: total revenue 2023

AdvancedRAG

### Total Revenue for 2023

According to the provided context, the total revenue for the year ended April 30, 2023, was $1,068,989,000. This information can be found several times in the provided text.         

SimpleRAG

According to the context provided, the total revenue for the fiscal year ended April 30, 2023, is:

$1,068,989,000 (or approximately $1.1 billion).        

Summary: Both RAGs got the right answer: $1,068,989,000 total revenue in 2023

Both of them were right here. It does seem like AdvancedRAG may have acquired a broader range of documents? Certainly the answer is more detailed and incorporates information from previous years.

That is to be expected given the enhancements we made, but it's far too early to call.

Let's raise the difficulty.

Test 3: What product does growth primarily depend on? How much?

AdvancedRAG

### Product Growth Primarily Depends On:

According to the context, "our revenue is derived primarily from subscriptions of search, observability and security built into the elastic stack." The document mentions various strategies and factors contributing to the company’s growth which include investment in "elastic cloud," "elastic stack," and the conversion of free users to paid subscribers.

### Quantitative Data:

- Revenue: The document states, "Our total revenue was $1.1 billion, 

$862.4 million, and $608.5 million for the years ended April 30, 2023, 

2022, and 2021, respectively."

- Subscription Revenue: "Subscriptions accounted for 92%, 93%, 

and 93% of our total revenue for the years ended April 30, 2023, 

2022, and 2021, respectively."

- Elastic Cloud Contribution: "For the years ended April 30, 2023, 

2022, and 2021, Elastic Cloud contributed 40%, 35%, and 27% of our 

total revenue, respectively."

- Customer Growth: The number of customers increased from 

"over 15,000 as of April 30, 2021," to "approximately 20,200 as of 

April 30, 2023."

### Summary:

Product growth for the company primarily depends on the "Elastic Stack" (which includes search, observability, and security solutions) and particularly the "Elastic Cloud." The Elastic Cloud's growing contribution to revenue highlights its significance, which grew from 27% in 2021 to 40% in 2023.        

SimpleRAG

According to the provided context, growth primarily depends on Elastic Cloud, which is described as the company's "family of cloud-based offerings." The context also emphasizes the significance of Elastic Cloud in achieving the company's long-term growth potential and its role in increasing subscription revenue.        

Summary: Both RAGs correctly identified Elastic Cloud as the key growth driver. However, AdvancedRAG includes more detail, factoring in subscription revenues and customer growth, and explicitly mentions other Elastic offerings.

Test 4: Describe employee benefit plan

AdvancedRAG

### Employee Benefit Plan Overview

The provided context outlines various elements of the employee benefit plan at Elastic, often referred to as "Elasticians." Here are the key components described in the texts:

1. 401(k) Plan:

   - Coverage: The 401(k) plan is available to substantially all U.S. employees who meet minimum age and service requirements.
...

2. Stock-Based Compensation:
...

3. Total Rewards Compensation:
...

4. Retirement and Income Protection Plans:
...

5. Flexible Work Environment: 
...

6. Learning and Development:
...

7. Employee Engagement and Surveys:
...

8. Onboarding Program:
...

In summary, Elastic's employee benefit plan encompasses a broad spectrum of financial, health, well-being, and professional development benefits designed to support and enhance the lives of its employees. The plan is both competitive and comprehensive, aiming to attract, retain, and engage its global workforce.        

SimpleRAG

The employee benefit plan at the Company includes several key components aimed at enhancing the overall well-being and professional development of employees, referred to as "Elasticians." 

1. Compensation and Benefits:
...

2. Retirement and Income Protection Plans:
...

3. Work Environment:
...

4. Community Involvement:
...        

Summary: AdvancedRAG goes into much greater depth and detail, mentioning the 401K plan for US-based employees, as well as defining contribution plans outside of the US. It also mentions Health and Well-Being plans but misses the Elastic Cares program, which SimpleRAG mentions.

Test 5: Which companies did Elastic acquire?

AdvancedRAG

Elastic acquired several companies over the years to enhance its technology and market presence. According to the provided context, the specific acquisitions include:

1. CmdWatch Security Inc.: Acquired on September 17, 2021. 
The total purchase consideration was $77.8 million.

2. Build Security Ltd.: Acquired on September 2, 2021. The combined purchase price for Build Security Ltd. and another acquisition was $57.2 million.

3. Optimyze Cloud Inc.: Acquired on November 1, 2021. Part of the combined purchase price mentioned above with Build Security Ltd.

4. Endgame, Inc.: Although the context mentions an amended and restated stock incentive plan, it indicates that Endgame, Inc. was associated with Elastic, suggesting it could have been acquired at some point.

These acquisitions are part of Elastic's strategy to enhance the technology underlying its security and observability offerings.        

SimpleRAG

The provided context does not specify any companies that Elastic has acquired. 

Therefore, based on the context, there is no information available about the companies acquired by Elastic.        

Summary: SimpleRAG does not retrieve any relevant info about acquisitions, leading to a failed answer. AdvancedRAG correctly lists CmdWatch, Build Security, and Optimyze, which were the key acquisitions listed in the report.

Conclusion

Based on our tests, our advanced techniques appear to increase the range and depth of the information presented, potentially enhancing quality of RAG answers. Additionally, there may be improvements in reliability, as ambiguously worded questions such as Which companies did Elastic acquire? and Who audits Elastic were correctly answered by AdvancedRAG but not by SimpleRAG.

However, it is worth keeping in perspective that in 3 out of 5 cases, the basic RAG pipeline, incorporating Hybrid Search but no other techniques, managed to produce answers that captured most of the key information.

We should note that due to the incorporation of LLMs at the data preparation and query phases, the latency of AdvancedRAG is generally between 2-5x larger that of SimpleRAG. This is a significant cost which may make AdvancedRAG suitable only for situations where answer quality is prioritized over latency.

The significant latency costs can be alleviated using a smaller and cheaper LLM like Claude Haiku or GPT-4o-mini at the data preparation stage. Save the advanced models for answer generation.

This aligns with the findings of Wang et al. As their results show, any improvements made are relatively incremental. In short, simple baseline RAG gets you most of the way to a decent end-product, while being cheaper and faster to boot.

For me, it's an interesting conclusion. For use cases where speed and efficiency are key, SimpleRAG is the sensible choice. For use cases where every last drop of performance needs squeezing out, the techniques incorporated into AdvancedRAG may offer a way forward.