Text Summarization With Deep Learning; LoftG: LoRA Fine-Tuning-Aware-Quantization; Generalizable CoT; Midjourney 6 VS. DALL-E 3; and More.

Editor's Paper Recommendations

Surveying the Landscape of Text Summarization with Deep Learning: A Comprehensive Review: In recent years, deep learning has revolutionized natural language processing (NLP) by enabling the development of models that can learn complex representations of language data, leading to significant improvements in performance across a wide range of NLP tasks. Deep learning models for NLP typically train deep neural networks with large amounts of data, allowing them to learn the patterns and relationships in language data. This contrasts with traditional NLP approaches, which rely on hand-engineered features and rules to perform NLP tasks. The ability of deep neural networks to learn hierarchical representations of language data, handle variable-length input sequences, and perform well on large datasets makes them well-suited for NLP applications. Text summarization has been a crucial research area in NLP due to the exponential growth of textual data and the rising demand for condensed, coherent, and informative summaries. Applying deep learning to text summarization involves using deep neural networks to perform tasks. This survey reviews fashionable text summarization tasks in recent years, including extractive, abstractive, multi-document, etc. Next, we discuss the most deep learning-based models and their experimental results on these tasks. The paper also covers datasets and data representation for summarization tasks. Finally, we delve into the opportunities and challenges associated with summarization tasks and their corresponding methodologies, aiming to inspire future research efforts to advance the field. Our survey aims to explain how these methods differ in their requirements, as understanding them is essential for choosing a technique suited for a specific setting.

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models: Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. This work focuses on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In these situations, there is often a steady difference in how well downstream tasks are done between full fine-tuning and quantization plus the LoRA fine-tuning approach. As a response, we came up with LoftQ (LoRA-Fine-Tuning-aware Quantization). This new quantization framework quantizes an LLM and finds the right low-rank initialization for LoRA fine-tuning simultaneously. This initialization eliminates the difference between the quantized and full-precision models and makes generalization much better in tasks that come after. We evaluate our method on natural language understanding, question answering, summarization, and generation tasks. Experiments have shown that our method performs significantly better than other quantization techniques, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. We will release our code.

Meta-CoT: Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models: Chain-of-thought (CoT) prompting, which creates intermediate reasoning chains that serve as the rationale for arriving at the answer, has shown that large language models (LLMs) can do amazing things with reasoning. However, current CoT methods employ general prompts such as Let's Think Step by Step or heavily rely on handcrafted task-specific demonstrations to attain preferable performances, engendering an inescapable gap between performance and generalization. To bridge this gap, we propose Meta-CoT, a generalizable CoT prompting method in mixed-task scenarios where the type of input questions is unknown. Meta-CoT first categorizes the scenario based on the input question and subsequently constructs diverse demonstrations from the corresponding data pool in an automatic pattern. Meta-CoT simultaneously performs remarkably well on ten public benchmark reasoning tasks and has superior generalization capabilities. Meta-CoT achieves the state-of-the-art result on SVAMP (93.7%) without any additional program-aided methods. Our further experiments on five out-of-distribution datasets verify the stability and generality of Meta-CoT.

InstructRetro: Instruction Tuning post-Retrieval-Augmented Pretraining: By using outside databases, pretraining auto-regressive large language models (LLMs) with retrieval shows better perplexity and factual accuracy. However, the current pre-trained retrieval-augmented LLM is still small (for example, Retro only has 7.5B parameters), which makes instruction tuning and zero-shot generalization less useful. This work introduces Retro 48B, the largest LLM pre-trained with retrieval before instruction tuning. Specifically, we continue to pre-train the 43B GPT model on an additional 100 billion tokens using the Retro augmentation method by retrieving 1.2 trillion tokens. The obtained foundation model, Retro 48B, largely outperforms the original 43B GPT regarding perplexity. After instruction tuning on Retro, InstructRetro significantly improves over the instruction-tuned GPT on zero-shot question-answering (QA) tasks. Specifically, the average improvement of InstructRetro is 7% over its GPT counterpart across 8 short-form QA tasks and 10% over GPT across 4 challenging long-form QA tasks. Surprisingly, one can ablate the encoder from the InstructRetro architecture and directly use its decoder backbone while achieving comparable results. We hypothesize that pretraining with retrieval makes its decoder good at incorporating context for QA. Our results highlight the promising direction to obtain a better GPT decoder for QA through continued pretraining with retrieval before instruction tuning.

--

Are you looking to advertise a product, job opening, or event to an audience of over 40,000 AI researchers and engineers? Please reach out to us on?LinkedIn?to explore your options.

Enjoy the newsletter? Help us make it bigger and better by sharing it with colleagues and friends.

--

Industry Insights

• Midjourney V6 is here with in-image text and completely overhauled prompting
• Midjourney 6 vs. Dall-E 3 prompt comparisons
• Reduce the costs of GPT-4 with prompt compression
• Claude AI Creator Anthropic to Raise $750 Million in New Funding: Report
• ?How to Access and Use Gemini API for Free

?

Growth Zone

• To Cope with Stress, Try Learning Something New
• How to Help Superstar Employees Fulfill Their Potential

?

Expert Advice