Retrieval-Augmented Language Models: Enhancing Knowledge and Factual Accuracy (Summarizing selected Research Paper on RAG)

In the ever-evolving landscape of natural language processing (NLP), researchers are continuously pushing the boundaries of what is possible. Two groundbreaking studies have introduced innovative approaches to augmenting large language models with external knowledge retrieval capabilities, paving the way for more accurate and informative language generation.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

A team of researchers from Facebook AI Research, University College London, and New York University introduced Retrieval-Augmented Generation (RAG) models (Reference Paper Summarized) , a novel framework that combines the power of pre-trained parametric models with non-parametric memory from Wikipedia. (Interactive Demo: HuggingFace)

The RAG models address a key limitation of large language models: their difficulty in accurately accessing and manipulating knowledge. By merging a pre-trained sequence-to-sequence model (such as BART) with a dense vector index of Wikipedia, accessed by a neural retriever, RAG models can generate more factual and knowledge-rich text.

Two variants of RAG were introduced:

RAG-Sequence, which uses the same retrieved document for the entire sequence, and

RAG-Token, which allows for different passages to be used for each token.

The retrieval component, called Dense Passage Retriever (DPR), employs a bi-encoder architecture with BERT-based document and query encoders.The generator component utilizes BART-large, a pre-trained seq2seq transformer with 400M parameters.

In open-domain question answering tasks, RAG models established new state-of-the-art results, outperforming both parametric sequence-to-sequence models and task-specific retrieve-and-extract architectures. Remarkably, RAG models demonstrated the ability to generate correct answers even when the right answer wasn't present in any retrieved document.

The RAG models employ a novel training approach that jointly optimizes the retriever and generator components in tandem. Unlike traditional methods that require explicit supervision on which documents to retrieve, RAG models learn this capability through the training process itself. The models leverage the vast knowledge contained within Wikipedia as their non-parametric memory bank, with the entire corpus split into 21 million easily retrievable chunks of 100 words each. This training strategy, powered by the stochastic gradient descent optimization algorithm called Adam, enables the RAG framework to seamlessly integrate retrieval and generation, unlocking new possibilities for knowledge-driven language understanding and generation.

Summarizing the various components of RAG Architecture as discussed in the paper:

A. Query and Document Embedding:

• The retrieval component is called Dense Passage Retriever (DPR) and uses a bi-encoder architecture.
• DPR employs BERT models for both the document encoder (BERTd) and query encoder (BERTq).
• For a document z, BERTd produces a dense vector representation d(z).
• For a query x, BERTq produces a query representation q(x).
• The document and query embeddings are created such that relevant documents for a given query are positioned close together in the vector space.
• This proximity of relevant document embeddings to the query embedding enables effective retrieval via maximum inner product search.

B. Retrieval Process:

• Retrieval involves calculating top-k documents with highest similarity to query embedding
• This is a Maximum Inner Product Search (MIPS) problem. MIPS problem is solved approximately in sub-linear time
• Allows efficient retrieval of most relevant documents from large knowledge base

C. End-to-end architecture:

• RAG uses input x to retrieve relevant documents z from knowledge base, Retrieved documents z provide additional context for generating target y.
• Generator is BART-large, a large pre-trained sequence-to-sequence transformer. BART-large combines input x and retrieved content z for generation.
• The RAG-Sequence model uses the same retrieved document for generating the complete sequence, while the RAG-Token model can use different passages per token.
• Retriever and generator components jointly trained without supervision on which documents to retrieve. Training optimizes negative log-likelihood of targets via stochastic gradient descent. The document encoder BERTd kept fixed during training to avoid reindexing and periodic updates of the document index.

D. Performance & Observations:

• RAG models achieved new state-of-the-art results on open-domain QA tasks. Outperformed parametric seq2seq models and retrieve-and-extract architectures. It could generate correct answers even without relevant retrieved documents.
• RAG-Sequence surpassed BART on Open MS-MARCO natural language generation. Showed reduced hallucinations and improved factual correctness.
• RAG-Token outperformed RAG-Sequence on Jeopardy question generation. Demonstrated higher factuality and answer specificity.
• On FEVER fact verification, RAG models matched state-of-the-art despite using simpler architectures without intermediate retrieval supervision.
• This study highlights effectiveness of hybrid generation models , combining parametric and non-parametric memories, thus opens new directions for NLP tasks.