Step-by-Step Guide to Creating Enterprise AI Chatbot Using RAG and Reranking

Introduction

Enterprises are fast realizing that putting customers at the heart of their strategy is not just a growth driver – it's essential for survival. This means prioritizing quick and efficient resolution of customer queries. Typically, companies achieve this by leveraging emails and support documentation, or by creating call centers. However, customers often prefer interactive forms of resolution over reading help and support documentation. Call centers are expensive to set up and operate, and in many cases, they are inundated with queries that could have been resolved without human intervention.?

Let’s talk data: McKinsey’s 2023 study reveals that 50% of customer queries can be reduced through AI-driven automation, freeing support executives to tackle more complex challenges. With 5,000 support executives, one company experienced a 14% boost in issue resolution per hour and cut handling time by 9% using this technology. Even more striking, support executive turnover and escalations to managers plummeted by 25%. With the potential to increase productivity by 30-45%, generative AI could be the key for enterprises seeking to drive top-line growth through AI-powered chatbots.

With the emergence of large language models (LLMs) and architectures like Retrieval-Augmented Generation (RAG), we now have the technology to create AI-powered chatbots that can reduce the burden on support executives. With LLM’s natural language understanding (NLU) and natural language generation capabilities, enterprises can create chatbots that respond in a human-like manner, leveraging knowledge from a company’s knowledge base. These chatbots can either be customer-facing or can even serve as assistants to support executives.?

In this tutorial, we'll guide you through the process of creating a powerful chatbot using RAG architecture enhanced with Reranking for improved performance and accurate query responses.

Understanding Retrieval-Augmented Generation With Reranking

RAG is a technique that improves the performance of large language models (LLMs). LLMs use their training data to generate a response, and therefore, they can hallucinate when presented with data they have not been trained on. Using RAG architecture, you can present contextual data to LLMs from a knowledge base outside their training data. This allows the LLM to generate responses based on that context, and go beyond their initial training data. RAG systems, therefore, are extremely powerful in extending the capabilities of LLMs to domains where enterprise data is involved.?

Basic RAG systems can sometimes struggle to generate accurate responses. This happens because RAG systems fetch a list of top results from the data source (vector database) based on their similarity with a query, but lack the full query context. Reranking improves RAG’s quality of retrieved documents by creating a query and document pair and assigning a similarity score to their output. It reorders them for contextual relevance based on these scores before presenting the documents to the LLM.?

In our chatbot, we will use the RAG system to first retrieve a set of potentially useful documents using methods like BM25 or dense vector search. We will then rerank these documents based on their relevance to the user's query. This involves assigning a new relevance score and reordering the documents accordingly. Finally, our system will select the top-ranked documents to inform the chatbot's responses, ensuring they are more accurate and contextually relevant.

The Architecture of RAG + Reranking Chatbot

Let’s look at the technology architecture we will be using.??

Llama 3.1: Llama 3.1 is a powerful and accessible large language model for building generative AI applications. Currently, it surpasses all the prevalent open-source LLMs and comes close in performance to the proprietary models. It is available in three sizes (405B, 70B, and 8B). Depending on the computational requirements and performance, you can decide which one to use. It offers superior accuracy and contextual awareness, making it ideal for complex tasks such as conversational AI, content generation, and language translation.??

Jina Reranker: Jina Reranker v2 is a neural model designed to refine search results in Retrieval-Augmented Generation (RAG) systems by reordering retrieved documents based on relevance. It supports multilingual text, structured data querying, and function calling, making it ideal for complex search tasks. Optimized with Flash Attention, it processes large inputs (up to 8192 tokens) and offers up to 6x speed improvements over its predecessor, making it suitable for real-time applications. The model integrates well with LLMs, and enhances retrieval accuracy and relevance.

E2E Cloud: Your AI-Focused Hyperscaler from India

E2E Cloud is a leading AI-centric hyperscaler offering advanced cloud GPUs like H100, A100, L4OS, and A40 at competitive rates. Our platform supports seamless integration and deployment of AI models, making it the preferred choice for private and public sector organizations, institutes, and universities. With TIR, our advanced AI development platform, we eliminate infrastructure challenges in the AI development process, ensuring efficient workflows. As a MeitY-empaneled service provider, E2E Cloud complies with regulations and IT laws, helping you build data-sovereign architectures.

?

Step-by-Step Guide to Building the AI Chatbot

Let’s start.

First, sign in or create an account on E2E MyAccount. After registering, you can launch a cloud GPU node. You'll need a GPU with over 32 GB of RAM. Alternatively, you can use a quantized version of the Llama 3.1 model from Hugging Face, which offers lower accuracy but requires less RAM.

For this experiment, we recommend using an A100 or H100 Cloud GPU. As your project scales, you can upgrade to InfiniBand-powered cluster configurations, such as 4xH100, 8xH100, or even 64xH100, for enhanced performance.

Once your cloud GPU is set up and your public SSH key is added, you're ready to follow the steps outlined below.

The following diagram illustrates our implementation architecture.

Now, we can create a Python virtual environment. We will also install and start Jupyter Lab.?

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install jupyterlab
$ jupyter lab
        

This will launch Jupyter Lab on your node. You can then create an SSH tunnel to access it remotely on your local machine browser.

$ ssh -L 8888:localhost:8888 developer@remote_host
        

Step 2: Install required libraries

Now we can install the required libraries in the Jupyter Lab environment.?

!pip install -q langchain-community gradio qdrant_client sentence-transformers
        

Let’s import the necessary libraries.

import pandas as pd
from sentence_transformers import SentenceTransformer
from langchain.embeddings import HuggingFaceEmbeddings
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
        

Step 3: Download Dataset

We will use an Indian financial news dataset from Hugging Face.

dataset =? "hf://datasets/kdave/Indian_Financial_News/training_data_26000.csv"?
        

Let’s download this and save it in a variable.?

df = pd.read_csv(dataset)

def download_dataset(df):
    count=0
    data=[]
    for content in df['Content']:
        temp=[]
        temp.append(content)
        temp.append(df["URL"][count])
        count=count+1
        data.append(temp)
    return data
    
data = download_dataset(df)
        

You can check the data in the following way:?

print(data[0])
print(len(data))
        

You should get something like this:?

['US consumer spending dropped by a record in April as the COVID-19 pandemic undercut demand, buttressing expectations that the economy could contract in the second quarter at its steepest pace since the Great Depression.\n\nThe Commerce Department said on Friday consumer spending, which accounts for more than two-thirds of U.S. economic activity, plunged 13.6 percent last month. That was the biggest drop since the government started tracking series in 1959, and followed a 6.9 percent tumble in March.\n\nEconomists polled by Reuters had forecast consumer spending plummeting 12.6 percent in April.\n\nFollow our full coverage of the coronavirus pandemic here.', 'https://www.moneycontrol.com/news/business/economy/covid-19-pandemic-us-consumer-spending-tumble-in-april-5334861.html']
26961
        

Step 4: Generate Embeddings and Store in Vector Store

We will now generate embeddings from this data and store them in the vector store. The choice of vector store is yours. Here, we will use Qdrant. You can use TIR to get a Qdrant endpoint, or install it using Docker. Let’s do the latter.?

$ sudo docker pull qdrant/qdrant
$ sudo docker run -p 6333:6333 -p 6334:6334 \
   -v $(pwd)/qdrant_storage:/qdrant/storage:z \
   qdrant/qdrant
        

Now that you have Qdrant running, you can generate embeddings and store them along with the payload in the following way.?

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

qd_client = QdrantClient(url="https://localhost:6333")

qd_client.recreate_collection(
    collection_name="fin_news",
    vectors_config=VectorParams(
        size=encoder.get_sentence_embedding_dimension(),
        distance=Distance.COSINE,
    ),
)

qd_client.upload_points(
    collection_name="fin_news",
    points=[
        PointStruct(
            id=idx, vector=encoder.encode(doc[0]).tolist(), payload={'id': idx, 'content': doc[0], 'url': doc[1]}
        )
        for idx, doc in enumerate(data)
    ],
)
        

This will take some time, so brew a cup of coffee. Once this is completed, you should have the vector store populated with your embeddings data and payload. You can verify this.?

results = qd_client.query_points(
    collection_name="fin_news",
    query=encoder.encode("US consumer spending").tolist(),
    limit=10,
).points

for result in results:
    print(result.payload, "score:", result.score)
        

This should return something like this:?

{'id': 0, 'content': 'US consumer spending dropped by a record in April as the COVID-19 pandemic undercut demand, buttressing expectations that the economy could contract in the second quarter at its steepest pace since the Great Depression.\n\nThe Commerce Department said on Friday consumer spending, which accounts for more than two-thirds of U.S. economic activity, plunged 13.6 percent last month. That was the biggest drop since the government started tracking series in 1959, and followed a 6.9 percent tumble in March.\n\nEconomists polled by Reuters had forecast consumer spending plummeting 12.6 percent in April.\n\nFollow our full coverage of the coronavirus pandemic here.', 'url': 'https://www.moneycontrol.com/news/business/economy/covid-19-pandemic-us-consumer-spending-tumble-in-april-5334861.html'} score: 0.74790907
{'id': 564, 'content': 'US consumer spending dropped by a record in April as the COVID-19 pandemic undercut demand, buttressing expectations that the economy could contract in the second quarter at its steepest pace since the Great Depression.\n\nThe Commerce Department said on Friday consumer spending, which accounts for more than two-thirds of U.S. economic activity, plunged 13.6 percent last month. That was the biggest drop since the government started tracking series in 1959, and followed a 6.9 percent tumble in March.\n\nEconomists polled by Reuters had forecast consumer spending plummeting 12.6 percent in April.\n\nFollow our full coverage of the coronavirus pandemic here.', 'url': 'https://www.moneycontrol.com/news/business/economy/covid-19-pandemic-us-consumer-spending-tumble-in-april-5334861.html'} score: 0.74790907
{'id': 5704, 'content': 'US consumer spending dropped by a record in April as the COVID-19 pandemic undercut demand, buttressing expectations that the economy could contract in the second quarter at its steepest pace since the Great Depression.\n\nThe Commerce Department said on Friday consumer spending, which accounts for more than two-thirds of U.S. economic activity, plunged 13.6 percent last month. That was the biggest drop since the government started tracking series in 1959, and followed a 6.9 percent tumble in March.\n\nEconomists polled by Reuters had forecast consumer spending plummeting 12.6 percent in April.\n\nFollow our full coverage of the coronavirus pandemic here.', 'url': 'https://www.moneycontrol.com/news/business/economy/covid-19-pandemic-us-consumer-spending-tumble-in-april-5334861.html'} score: 0.74790907
…
        

Now that we have our vector data sorted, let’s look at reranking.?

Step 5 - Reranking Using Jina Reranker V2

Reranking plays a critical role in improving Retrieval-Augmented Generation (RAG) systems by enhancing the precision of the search results returned from initial retrieval. In a typical RAG workflow, you first retrieve a set of documents based on their vector similarity to a query.?

However, this initial step often lacks contextual depth, especially when dealing with ambiguous or complex queries. This is where reranking comes in. It reorders these retrieved documents based on their relevance scores, computed using cross-encoders or other neural models, analyzing the relationships between the query and every document.?

By reranking, you ensure that the most contextually appropriate documents are sent to the language model for further processing, which significantly boosts the overall quality and relevance of generated outputs.

In our RAG workflow, we already have the vector results being returned. However, we will additionally use reranking to improve the ordering before prompting the LLM.?

Here’s how you can rerank the results retrieved in the previous step.?

from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "jinaai/jina-reranker-v2-base-multilingual",
    automodel_args={"torch_dtype": "auto"},
    trust_remote_code=True,
)

# Query
query = "US consumer spending"

# Content from results retrieved above
documents = []

for result in results:
    documents.append(result.payload['content'])

# construct sentence pairs
sentence_pairs = [[query, doc] for doc in documents]

scores = model.predict(sentence_pairs, convert_to_tensor=True).tolist()

rankings = model.rank(query, documents, return_documents=True, convert_to_tensor=True)

print(f"Query: {query}")
for ranking in rankings:
    print(f"ID: {ranking['corpus_id']}, Score: {ranking['score']:.4f}, Text: {ranking['text']}")
        

You will find that the reranker has rescored the documents and returned a list that’s more relevant.?

Step 6: RAG + Reranking

Now let’s connect it all together. We will instantiate the LLM and prompt it with the retrieved context.?

To do this, let’s use OpenLLM, which allows us to serve Llama3.1-8b locally using an OpenAI compatible API.?

First, on your node, install OpenLLM and set it up to serve Llama3.1-8b-Instruct.

$ pip install openllm
$ openllm serve llama3:8b
        

At this point you will have Llama3.1-8b running locally. You can test it by simply using OpenAI APIs.?

from openai import OpenAI

client = OpenAI(base_url='https://localhost:3000/v1', api_key='na')

# Use the following func to get the available models
# model_list = client.models.list()
# print(model_list)

chat_completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Explain the meaning of consumer spending"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    print(chunk.choices[0].delta.content or "", end="")
        

Now let’s prompt the LLM with our reranked documents as context, and generate responses to our queries.?

First, we create the context for the LLM from the reranked responses.?

llm_context = []
for ranking in rankings:
    llm_context.append(ranking['text'])
final_context = '\n'.join(llm_context)
print(final_context)
        

Now, let’s prompt the LLM:?

question = "Did Covid-19 lead to a dip in US consumer spending?"

chat_completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {
            "role": "user",
                "content": f"This is the question asked by user {question} and the context given is {final_context} answer this question based on the context provided",
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    print(chunk.choices[0].delta.content or "", end="")
        

This will give you an accurate response to the query:?

Yes, COVID-19 led to a dip in US consumer spending. According to the article, US consumer spending dropped by a record 13.6% in April, which was the biggest drop since the government started tracking series in 1959. This decline in consumer spending is attributed to the COVID-19 pandemic, which undercut demand and led to a contraction in the economy.
        

Yes, COVID-19 led to a dip in US consumer spending. According to the article, US consumer spending dropped by a record 13.6% in April, which was the biggest drop since the government started tracking series in 1959. This decline in consumer spending is attributed to the COVID-19 pandemic, which undercut demand and led to a contraction in the economy.

You now have the foundation of a system that uses vector store, reranker and Llama3.1-8B to create a powerful enterprise AI system.?

Next Steps

In this guide, we looked at how to pull together embedding generation, reranking, and Llama3.1-8b to create an enterprise AI system. With the right dataset and a data sovereign cloud like E2E Cloud, you can build an AI system within 24-48 hours.?