Top LLM Papers of the Week (July Week-1 2024)

[1] Scaling Synthetic Data Creation with 1,000,000,000 Personas

This paper introduces a novel persona-driven data synthesis methodology that leverages various perspectives within a large language model (LLM) to create diverse synthetic data. The authors introduce Persona Hub – a collection of 1 billion diverse personas automatically curated from web data. (Paper)

[2] XTOWER: A Multilingual LLM for Explaining and Correcting Translation Errors

While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies. This paper introduces XTOWER, an open large language model (LLM) built on top of TOWERBASE designed to provide free-text explanations for translation errors in order to guide the generation of a corrected translation. (Paper)

[3] BERGEN: A Benchmarking Library for Retrieval-Augmented Generation

Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. This paper introduces BERGEN, an end-to-end library for reproducible research standardizing RAG experiments. (Paper)

[4] Retrieval-augmented generation in multilingual settings

Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. This paper explores RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages. (Paper)

[5] From RAG to RICHES: Retrieval Interlaced with Sequence Generation

This paper introduces RICHES, a novel approach that interleaves retrieval with sequence generation tasks. RICHES offers an alternative to conventional RAG systems by eliminating the need for separate retriever and generator. It retrieves documents by directly decoding their contents, constrained on the corpus. (Paper)

For NLP Research and NLP Project guidance, please check

[6] Pistis-RAG: A Scalable Cascading Framework Towards Content-Centric Retrieval-Augmented Generation

This paper introduces Pistis-RAG, a scalable multi-stage framework designed to address the challenges of large-scale retrieval-augmented generation (RAG) systems. This framework consists of distinct stages: matching, pre-ranking, ranking, reasoning, and aggregating. (Paper)

[7] RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). This work presents RankRAG, a novel instruction fine-tuning framework, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. Llama3-RankRAG (i) significantly outperforms Llama3-ChatQA-1.5 and GPT-4 models on nine knowledge-intensive benchmarks and (ii) performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning. (Paper)

[8] Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets

Most Instruction Fine-Tuning (IFT) datasets are predominantly in English, limiting model performance in other languages. This paper introduces a novel method for collecting multilingual IFT datasets that preserves linguistic naturalness and ensures prompt diversity. This approach leverages English-focused LLMs, monolingual corpora, and a scoring function to create high-quality, diversified IFT datasets in multiple languages. (Paper)

[9] How Does Quantization Affect Multilingual LLMs?

Quantization techniques are widely used to improve inference speed and deployment of large language models. This paper conducts a thorough analysis of quantized multilingual LLMs, focusing on their performance across languages and at varying scales. The authors observed that languages are disparately affected by quantization, with non-Latin script languages impacted worst. (Paper)

[10] Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application

Knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. (Paper)

If you like this, do subscribe to the newsletter so that you won't miss reading interesting LLM papers.