Top LLM Papers of the week (July Week 4, 2024)

[1] The Llama 3 Herd of Models

This paper introduces Llama 3.1, the SOTA open-source LLM. Llama 3.1 is available in three sizes namely 8B, 70B and 405B. Both pretrained and instruction-tuned variants are available for each of these. [Paper]

[2] OpenDevin: An Open Platform for AI Software Developers as Generalist Agents

This paper introduces OpenDevin, a platform for developing powerful and flexible AI agents that interact with the world in similar ways to those of a human developer: by writing code, interacting with a command line, and browsing the web. Released under the permissive MIT license, OpenDevin is a community project spanning academia and industry with more than 1.3K contributions from over 160 contributors. [Paper]

[3] Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

KV-Cache has emerged as a pivotal solution to handle long contexts in LLMs by converting the time complexity of token generation from quadratic to linear. This survey paper covers various properties of KV-Cache and elaborates on various methods currently used to optimize the KV-Cache space usage of LLMs. [Paper]

[4] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

This paper introduces Self-Route, a simple yet effective method that routes queries to RAG or LC (Long Context) based on model self-reflection. LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. [Paper]

[5] LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference

This paper introduces LazyLLM, a generic method that can be seamlessly integrated with existing language models to significantly accelerate the generation without fine-tuning. LazyLLM selectively computes the KV for tokens important for the next token prediction in both the prefilling and decoding stages. LazyLLM accelerates the prefilling stage of the Llama2 7B model by 2.34x while maintaining accuracy. [Paper]

For NLP Research and NLP Project guidance, please check

[6] A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More

This paper provides a comprehensive review of different LLM Alignment Techniques like RLHF, RLAIF, PPO, DPO etc. This paper categorizes papers into distinct topics and provides detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field. [Paper]

[7] Falcon2-11B Technical Report

This paper introduces Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license. [Paper]

[8] An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought

This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. [Paper]

[9] A Survey on Employing Large Language Models for Text-to-SQL Tasks

Writing SQL queries requires specialized knowledge, which poses a challenge for non-professional users and Text-to-SQL parsing solves this issue by converting natural language queries into SQL queries. This paper provides a comprehensive overview of LLMs in text-to-SQL tasks, discussing benchmark datasets, prompt engineering, fine-tuning methods, and future research directions. [Paper]

[10] XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models

This paper provides a comprehensive overview of XAI's role in LLM research. [Paper]

If you like this, do subscribe to the newsletter so that you won't miss reading interesting LLM papers.