Embracing the Future of Natural Language Processing - RAG

Retrieval-Augmented Generation (RAG) systems are transforming how we approach NLP by seamlessly integrating retrieval and generation models. This innovative approach bridges the gap between factual accuracy and language fluency, opening new frontiers in dialogue systems, question answering, and content generation.

Let’s break down how RAG systems work using a simple example: Imagine a chatbot querying a private knowledge base using langchain (Orchestrator), AzureOpenAI for embeddings & GPT-4 models, and FAISS for local vector DB.

1?? Ingestion Phase: First, we store documents in the vector database by:

• Loading documents
• Splitting them into chunks
• Creating embeddings
• Storing these embeddings in the vector database

# load the document --> split the document into chunks --> create embeddings --> store in vector database

#Split the entire knowledge base or collection of documents in chunks, chunks basically represents the single piece of context to be queried.

 pdf_path = "Vectors-in-memory/2210.03629v3.pdf"
    loader = PyPDFLoader(file_path=pdf_path)
    documents = loader.load()
    text_splitter = CharacterTextSplitter(
        chunk_size=1000, chunk_overlap=30, separator="\n"
    )
    docs = text_splitter.split_documents(documents=documents)

# Use embedding model to transform each chunks into vector embeddings.
embeddings = AzureOpenAIEmbeddings(
    azure_deployment=os.environ["AZURE_EMBEDDING_OPENAI_DEPLOYMENT"],
    openai_api_key=os.environ["AZURE_EMBEDDING_OPENAI_KEY"],
    azure_endpoint=os.environ["AZURE_EMBEDDING_OPENAI_API_BASE"],
    openai_api_type=os.environ["AZURE_EMBEDDING_OPENAI_API_TYPE"],
    openai_api_version=os.environ["AZURE_EMBEDDING_OPENAI_API_VERSION"],
 )

#Store all the vector embeddings into the vector database.
vectorstore = FAISS.from_documents(docs, embeddings)
vectorstore.save_local("faiss_index_react")        

2?? Retrieval Phase: When a query (or chatbot question) comes in, the system retrieves relevant information by:

• Embedding the query
• Using Approximate Nearest Neighbor (ANN) to find similar embeddings in the vector DB
• Returning relevant contexts or vectors

#convert the query(text) into embedding --> pass it to vector db --> map the results (context) to the prompt --> pass the prompt (query + context) to LLM

# Embed the query(which is asked from the chatbot) using the same embedding model.  
chat = AzureChatOpenAI(
        openai_api_key="xxxxxxd8a815xxxxxxxxxxxxx",
        azure_endpoint="https://testmodel.openai.azure.com/",
        openai_api_type="azure",
        azure_deployment="GPT4",
        openai_api_version="2024-05-01-preview",
        temperature=0,
        max_tokens=None,
        timeout=None,
        max_retries=2,
    )
   #Let below be the query from the chatbot
   query = "What is machine learning?"

   new_vectorstore = FAISS.load_local(
      "faiss_index_react",embeddings,allow_dangerous_deserialization=True
   )

   template = """Use the following pieces of context to answer the question at the end.
    If you don't know the answer, just say that you don't know, don't try to make up the answer.
    Use three sentences maximum and keep the answer as concise as possible.
    Always say "Thank you for Asking!!!" at the end of the answer.

    {context}

    Question: {question}

    Helpful Answer:"""

  custom_rag_prompt = PromptTemplate.from_template(template)

#The retrieval aspect involves retrieving relevant information or contexts from a large #corpus of text based on a given query or input. This retrieval is typically performed #using information retrieval techniques, where documents or passages most relevant # to the input are selected.

  rag_chain = (
      {"context": new_vectorstore .as_retriever(), "question": RunnablePassthrough() }
      | custom_rag_prompt
      | chat
    )

    res = rag_chain.invoke(query)
    print(res)        

3?? Generation Phase: Finally, the retrieved information is combined with the query to generate contextually relevant responses using models like GPT-4.

The synergy of retrieval and generation in RAG systems ensures responses are not only fluent but also grounded in accurate knowledge from the retrieval process.

However, a critical consideration emerges: How eco-friendly is this computational process? As we advance in AI, sustainability becomes pivotal. Let’s discuss how we can innovate responsibly and sustainably in the world of AI and NLP.