Summary of STaR: Boosting AI Reasoning with Iterative Self-Learning Technique

I developed a custom GPT, accessible here, designed to generate skimmable summaries of research articles and transcripts. This tool is aimed at simplifying the process of digesting the main points of complex documents quickly. I'm curating a catalog of articles that I find intriguing, sharing these summaries for easy consumption.

Whenever possible, I will include links to the full articles. Please note, all summary content is AI-generated unless stated otherwise."

Link to full article:

https://research.google/pubs/star-self-taught-reasoner-bootstrapping-reasoning-with-reasoning/

Summary:

Introduction:

• Humans often make decisions through extended chains of thought, and recent studies show that explicit intermediate reasoning (rationales) can enhance large language models (LLMs) performance across complex tasks like math and commonsense question-answering.
• Traditional methods to induce rationale generation in models are either resource-intensive or compromise on accuracy. The paper introduces a technique leveraging a small set of rationale examples and a large non-rationale dataset to iteratively enhance complex reasoning capabilities, termed "Self-Taught Reasoner" (STaR).

Method:

• STaR's Loop:Generate Rationales: Use a few rationale examples to prompt the model for rationales on many questions. Corrective Rationalization: If answers are incorrect, generate new rationales using the correct answers. Fine-tune: Fine-tune the model on all rationales leading to correct answers. Repeat: Iterate the process to improve.
• Rationalization: For problems the model fails to solve, rationalization involves generating a rationale given the correct answer, aiding the model in "thinking backwards" to enhance reasoning.

Experiments:

• Domains: Tested on arithmetic, commonsense reasoning, and grade school math problems.
• Base Model: Utilized GPT-J (6B-parameter model) for experiments.
• Findings: STaR significantly enhances performance on various datasets, demonstrating the efficacy of the bootstrapping approach for improving reasoning capabilities of LLMs.

Here's how the information discussed can be utilized immediately by anyone with access to LLMs:

1. Few-Shot Prompting: Start with a small set of examples that demonstrate the desired reasoning process (e.g., step-by-step solutions for math problems or rationales for commonsense questions). Use these examples as few-shot prompts to elicit the LLM's existing reasoning abilities.
2. Rationale Generation: Prompt the LLM to generate rationales or step-by-step solutions for a large set of problems. Filter out the generated rationales that lead to incorrect answers.
3. Rationalization: For problems where the LLM failed to generate a correct rationale, provide the correct answer as a hint and prompt the LLM to generate a rationale that justifies that answer.
4. Fine-tuning: Fine-tune the LLM on the filtered set of correct rationales generated in steps 2 and 3. This allows the LLM to learn from its own reasoning processes.
5. Iterate: Repeat steps 2-4 using the fine-tuned LLM from the previous iteration. This iterative process helps the LLM bootstrap its reasoning abilities, gradually improving its performance on complex tasks.

Example Prompt:

Human: Here are a few examples of step-by-step solutions for arithmetic problems: 

<example 1> Question: 42 + 37 Solution: 42 + 37 = 40 + 30 + 2 + 7 (Split both numbers into tens and ones) = 70 + 9 (Add the tens and ones separately) = 79 (Combine the results) 
</example 1> 

<example 2> 
Question: 156 - 89 Solution: 156 - 89 = 150 - 80 + 6 - 9 (Split both numbers into hundreds, tens, and ones) = 70 - 3 (Subtract hundreds, tens, and ones separately) = 67 (Combine the results) 
</example 2>

Human: Now, generate a step-by-step solution for the following arithmetic problem:

LLM: Here is a step-by-step solution for the arithmetic problem 683 + 247:

683 + 247 = 600 + 80 + 3 + 200 + 40 + 7 (Split both numbers into hundreds, tens and ones) = 800 + 120 + 10 (Add the hundreds, tens and ones separately) = 900 + 30 (Combine the hundreds and tens) = 930 (Combine the final result)

Therefore, the solution to 683 + 247 is 930.        

Limitations and Impacts:

• The technique requires balancing between generating new rationales and ensuring the quality of reasoning, with potential risks including amplification of biases and unfaithful explanations due to model's internal processes.

Conclusion:

• STaR represents a novel method for iteratively improving a model's reasoning capabilities, demonstrating notable performance improvements across different reasoning tasks. Further research is needed to refine the approach, especially concerning biases and the faithfulness of generated rationales.

Additional References:

• Eric Zelikman on Google Scholar
• Jesse Mu on Google Scholar
• Noah D. Goodman on Google Scholar