A Deep Dive into Retrieval-Augmented Multi-modal Chain-of-Thought Reasoning

Introduction

The release of Google Gemini models last week created a buzz around Large Multi-Modal Models. A groundbreaking paper titled Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models also emerged last week, presenting a novel approach that significantly enhances the reasoning capabilities of Large Language Models (LLMs) in multi-modal tasks. This article is a gold mine if you are interested in RAGs for multi-modal scenarios. There is a lot to learn in this paper so let's unpack it.

Understanding the paper's context requires a look at In-Context Learning and Chain-of-Thought Reasoning:

1. In-Context Learning (ICL): This is a learning paradigm using LLMs to perform tasks based on examples provided in the immediate context. ICL has been effective in various NLP tasks, including complex tasks like mathematical reasoning. The paper discusses the sensitivity of ICL to settings such as prompt structure, in-context example selection, and their order. ICL's potential in other modalities, especially visual, is an area of active research and constantly evolving.
2. Chain-of-Thought Reasoning (CoT): CoT is a technique to enhance LLMs' reasoning capabilities by guiding them through the reasoning process step by step, either in a zero-shot or few-shot manner. CoT significantly improves LLMs' reasoning abilities across various tasks.

Background: The Challenge of Multi-Modal Reasoning

Multi-modal tasks, involving both text and images, pose a unique challenge in AI. Traditional LLMs, adept at processing text, often struggle with these complex tasks. The CoT reasoning method where models simulate human-like step-by-step reasoning, has shown promise but remained predominantly text-focused. This paper's breakthrough lies in extending CoT reasoning to multi-modal scenarios, integrating both visual and textual elements. The paper reviews different approaches within CoT, such as sampling multiple reasoning paths, partitioning complex problems into sub-problems, and dynamically selecting diverse demonstration examples for CoT prompting. It also discusses adapting CoT for different modalities, including multi-modal settings.

Novel Methodology

• Retrieval Mechanism: The paper introduces a dynamic retrieval mechanism that automatically selects optimal CoT demonstration examples from a pool based on the query's textual and visual context as well as cross-modal similarities. This means the model can intelligently choose examples that closely match the text and images in a given query, ensuring more relevant and contextual reasoning.

• Stratified Sampling Strategy: To address the diversity in multi-modal tasks, the authors propose a stratified sampling method. This technique categorizes demonstration examples into groups based on their content type: textual, visual, or both. This approach involves dividing the demonstration pool into two distinct groups: one containing only textual context (Q_txt) and the other including both visual and textual context (Q_img). By doing so, the paper aims to ensure that the LLMs receive a diverse set of demonstration examples, thus enhancing the robustness of multi-modal reasoning.

• Final Prediction: After retrieving the demonstration examples, they are combined with the test question to create an enriched context for the LLM. This comprehensive prompt, consisting of both the question and relevant examples, is then used to generate the final answer.

Example to Illustrate the Methodology

Imagine a scenario where a model is asked to determine the common characteristics of a cat and a tiger, given their images and descriptions. The retrieval mechanism would sift through a pool of examples and select those relevant to mammals, cats, or similar contexts. The stratified sampling ensures that both textual and visual aspects are considered, allowing the model to reason that both are mammals, have fur, and are carnivorous, providing a nuanced and accurate answer.

Dataset and Models

The experimentation was conducted using the ScienceQA dataset, a multi-modal dataset encompassing a range of scientific topics in text and image formats. This diversity provided an ideal testing ground for the paper's methodologies. Models used included ChatGPT (GPT-3.5-Turbo), GPT-4, and GPT-4V, with encoders like SentenceBERT for text, ViT(ViT-base-patch16-224) for images and for cross-modality similarity, texts and images were encoded using CLIP.

Ablation Study and Its Findings

The ablation study in the paper critically analyzed the impact of various components of the methodology. It revealed that Stratified Sampling outperforms Random Sampling, demonstrating the importance of structured example selection. Furthermore, different retrieval methods showed varied effectiveness based on the type of question and the number of demonstration examples used. Here's a summary of the key findings:

1. Stratified Sampling vs. Random Sampling: The study compared the effectiveness of Stratified Sampling (the proposed method) against Random Sampling (where demonstration examples are randomly selected from the whole pool). Results showed that Stratified Sampling yielded substantial improvements over Random Sampling across all categories of questions in the ScienceQA dataset. This highlights the importance of selecting demonstration examples in a more structured and informed manner to enhance the model's reasoning ability.
2. Different Retrieval Methods: The paper also explored the impact of different retrieval methods on model performance, including Text-to-Text, Text-to-Image, Image-to-Text, and Image-to-Image retrieval. The study examined these methods under varying numbers of demonstration examples (shots) to assess their impact on model performance. It was observed that Image-to-Image Retrieval showed a significant improvement in accuracy with increasing shots, indicating the effectiveness of this method in leveraging visual context. However, for other retrieval methods like Text-to-Text, the benefits seemed to plateau or slightly decrease with more shots, suggesting that more demonstrations do not always guarantee better performance.
3. Performance Across Question Types: The study also analyzed the performance of different retrieval methods across various question types in the ScienceQA dataset. This analysis provided insights into which methods work best for different types of questions, highlighting areas that might benefit from more targeted approaches or methodological refinements.

Conclusion: The Impact and Future Implications

This paper unlocks new insights into multi-modal reasoning. By integrating text and images in the CoT reasoning process and utilizing a nuanced approach to example selection, it sets a new direction in the field. The methodologies and findings not only enhance the capabilities of LLMs in complex multi-modal reasoning tasks, but the methodology is applicable for single modality reasoning problems as well.

Paper: 2312.01714v1.pdf (arxiv.org)

Contributors to the paper: Bingshuai Liu, Chenyang L. , Zijun Min, Zhanyu Wang, Jinsong Su, Longyue WANG