ReAct: Teaching AI to Think and Act Like Us (But for Real!)

The paper "ReAct: Synergizing Reasoning and Acting in Language Models" was published in ICLR 2023.

Paper URL:?https://arxiv.org/pdf/2210.03629

Disclaimer:?the opinions I share are solely my own and do not reflect those of my employer

Imagine asking your computer to plan a trip, write a story, or troubleshoot your Wi-Fi. That sounds cool. That's the promise of AI agents —computer programs that can follow instructions and reason, learn, and take action to achieve a goal.

One exciting way to build these agents is with the ReAct framework. Think of ReAct as giving AI a brain and a pair of hands, letting it both think through problems and act in the world to solve them.

What is ReAct Framework?

ReAct stands for Reasoning and Acting. It's a method that helps language models (the brains behind AI like ChatGPT) tackle complex tasks by combining two key abilities:

• Reasoning: Breaking down problems into smaller steps, making plans, and learning from mistakes.
• Acting: Interacting with the outside world to gather information or take action to achieve goals. This could mean searching the web, using a calculator, or even controlling a robot.

Instead of just spitting out an answer, ReAct encourages the AI to think out loud, explaining its reasoning step-by-step, and then take actions based on that reasoning. It then observes the results of its actions and uses that information to refine its thinking and adjust its plans.

Why it was created:

• Existing reasoning-based prompting solutions leverage only the internal knowledge of the models to generate reasoning steps.
• Existing action-based prompting solutions execute an action by leveraging only external knowledge to identify the responses to the problems without reasoning.
• Fine-tuning or reinforcement learning-based solutions require a workforce to create annotated datasets with reasoning trajectories.

How does ReAct Framework Works: A Simple Analogy

The ReAct framework helps computers think and act more like humans when solving problems. It's like giving a computer a brain that can both reason and take action to figure things out.

Here's how it works:

1. The computer gets a question or task. Imagine the task is: "Find a clean knife and put it on the kitchen counter".
2. The computer "thinks" about how to solve the problem. This is the "Reason" part of ReAct. The computer might think, "First, I need to find a knife. Then, I need to clean it. Finally, I need to put it on the counter".
3. The computer takes an action. This is the "Act" part. It could be something like "search for a knife in the kitchen". Or "go to the cabinet".
4. The computer sees what happens after its action. This is like getting an observation. If the computer searched for a knife, it might "see" a list of places where knives are usually found. Or the computer "sees" a knife on the countertop.
5. The computer uses what it sees to think again and plan its next action. The computer might now think, "Okay, I see a knife on the counter. Now I need to pick it up".
6. The computer keeps repeating steps 3-5 (Act-Observe-Reason) until the problem is solved. It's like a loop of thinking and acting.

How is the ReAct Framework different from the previous implementation?

• ReAct combines thinking and acting. Other methods only focus on one or the other. For example, Chain of Thought (CoT) is just thinking. It's like planning a trip without looking at a map or checking bus schedules. Act-only is just acting, like wandering around without a plan.
• ReAct can use new information to improve. If the computer makes a mistake or finds new info, it can change its plan.

Example: Let's say the computer plays a text-based game called ALFWorld, where it needs to find a key and open a door.

• Regular "Thinking" (CoT): The computer might just think, "I need to find a key and open the door." But it doesn't know where to look or what to do next. It's like trying to find something in your house without looking. The computer might start hallucinating.
• Regular "Acting":?The computer might just try random actions, like "go to the kitchen" or "open the fridge," without any plan. It's like wandering around your house, hoping to find your needs.
• ReAct: The computer thinks, "I need to find a key." Then it acts: "search the desk for a key." It observes: "I found a key on the desk!" Then it thinks again: "Now I need to open the door with the key." Then it acts. The computer then observes the door opening, and the task is completed.

Why is ReAct so cool?

• Reduces Hallucinations: ReAct is less likely to make stuff up by grounding its reasoning in real-world information.
• Handles Complex Tasks: ReAct can break down big problems into smaller, more manageable steps.
• More Human-Like: ReAct's reasoning process is more manageable for humans to understand and trust.
• Versatile: ReAct can be used for many tasks, from answering questions to playing games.

ReAct Agents in Real-time.

ReAct agents are available in both LangChain and LlamaIndex, and they function similarly in both frameworks, emphasizing a combination of reasoning and acting based on user inputs.

Implementing ReAct in LangChain:

• LangChain provides a structured way to define an agent's actions and how it will reason through different scenarios.
• You can set up a chain of actions that respond to user queries, allowing for flexible and dynamic interactions.

Implementing ReAct in LlamaIndex:

• LlamaIndex also supports ReAct-style agents by integrating reasoning with access to external knowledge sources.
• By utilizing the indexing capabilities, the ReAct agent can retrieve and synthesize information effectively to respond accurately to user requests.

Both frameworks leverage the ReAct philosophy to create intelligent agents that can reason through information and act accordingly. By integrating these concepts, developers can build sophisticated applications that enhance user interaction and provide valuable insights.

ReAct Agents Implementation in LangChain and LlamaIndex.

Here’s an end-to-end real-time example using LangChain and LlamaIndex to analyze data with the ReAct Agent. This example involves querying a dataset (for instance, product reviews from Amazon) using LangChain, processing the data with LlamaIndex, and finally obtaining insights through a chat interaction.

Step-by-Step Example

Could you make sure you have the necessary packages installed? You can install them using pip:

pip install langchain llama-index openai        

• Import Necessary Libraries: First, import the libraries you need.

from langchain import LLMChain
from langchain.agents import create_openai_functions_agent
from llama_index import QueryEngine, SimpleDocument
import openai
import os        

• Set Up OpenAI API Key: Ensure you have your OpenAI API key set up. You can get it from the OpenAI website and set it in your environment variable.

os.environ["OPENAI_API_KEY"] = "your_openai_api_key"        

• Define the Data: Let’s say you’re analyzing Amazon product reviews. You would create a simple dataset.

reviews = [
    SimpleDocument("I love this product! It works great and is very affordable."),
    SimpleDocument("The quality is terrible. It broke after one use."),
    SimpleDocument("Excellent customer service, but the product didn't meet my expectations."),
]        

• Create the Query Engine: Initialize the query engine using the provided dataset.

query_engine = QueryEngine(reviews)        

• Set Up the LLM Chain with LangChain: Create the language model chain for processing inputs.

llm = LLMChain.from_openai(model="gpt-3.5-turbo")
agent = create_openai_functions_agent(llm)        

• Define the ReAct Loop: Implement the reasoning and acting loop for the ReAct Agent.

def react_agent_interaction(input_text):
    # Step 1: Decide whether to use the query engine
    should_query = agent.predict(f"Should I query the data for: {input_text}?")
    
    if should_query.lower() == "yes":
        # Step 2: Query the query engine
        query_result = query_engine.query(input_text)
        
        # Step 3: Decide whether to repeat the process or respond
        response = agent.predict(f"Based on the result '{query_result}', what can you conclude?")
        return response
    else:
        return "No querying necessary."

Execute the Interaction: Now that everything is set up, you can test the agent with an input.
user_input = "What do people think about the product quality?"
response = react_agent_interaction(user_input)
print("Agent Response:", response)        

Full Example Code

Putting it all together gives you the complete code below:

from langchain import LLMChain
from langchain.agents import create_openai_functions_agent
from llama_index import QueryEngine, SimpleDocument
import openai
import os

# Set OpenAI API key
os.environ["OPENAI_API_KEY"] = "your_openai_api_key"

# Define a simple dataset of reviews
reviews = [
    SimpleDocument("I love this product! It works great and is very affordable."),
    SimpleDocument("The quality is terrible. It broke after one use."),
    SimpleDocument("Excellent customer service, but the product didn't meet my expectations."),
]

# Create the query engine
query_engine = QueryEngine(reviews)

# Set up the LLM chain and agent
llm = LLMChain.from_openai(model="gpt-3.5-turbo")
agent = create_openai_functions_agent(llm)

def react_agent_interaction(input_text):
    should_query = agent.predict(f"Should I query the data for: {input_text}?")
    
    if should_query.lower() == "yes":
        query_result = query_engine.query(input_text)
        response = agent.predict(f"Based on the result '{query_result}', what can you conclude?")
        return response
    else:
        return "No querying necessary."

# Execute the interaction
user_input = "What do people think about the product quality?"
response = react_agent_interaction(user_input)
print("Agent Response:", response)        

Explanation

• This code initializes a simple dataset of product reviews and sets up the necessary components to create a ReAct Agent using LangChain and LlamaIndex.
• The agent processes inputs and provides responses based on whether it needs to query the dataset.
• You can modify the reviews array to analyze different data as needed.

How it is used in real-time use cases:

• Shopping agents: ReAct can be used as a shopping agent because one-shot Act prompting already performs on par with Imitation Learning (IL) and Imitation Learning + Reinforcement Learning (IL + RL) methods on Webshop.
• Human-in-the-loop behavior correction: ReAct allows a human to inspect and edit reasoning traces, which can drastically change the model's behavior.
• Up-to-date knowledge retrieval: ReAct can retrieve up-to-date information from the Internet and provide a reasonable answer, which can benefit Internet-augmented language models for up-to-date task-solving.
• Webpage navigation: ReAct can be used for webpage navigation.
• Text-based games: ReAct can be used in text-based games.
• Complex environments: ReAct can be applied to interactive environments within a closed-loop system.

Alternative frameworks and prompting methods that can be used alongside or instead of ReAct

• Chain-of-Thought (CoT) prompting encourages LLMs to break down problems into a series of intermediate steps, which improves transparency and interpretability. The "Let's think step by step..." prompt is an example of zero-shot CoT.
• Self-Consistency with Chain-of-Thought (CoT-SC) involves generating multiple results from the same CoT prompt and selecting the majority answer to reduce hallucinations. The ReAct framework can be combined with CoT-SC, allowing the model to decide when to switch between the two methods.
• Act-only prompting involves removing the "thoughts" in ReAct trajectories. According to the "ReAct: Synergizing Reasoning and Acting in Language Models" paper, one-shot Act prompting performs on par with Imitation Learning (IL) and Imitation Learning + Reinforcement Learning (IL + RL) methods and can be used as a shopping agent.
• Language Agent Tree Search is mentioned as the subject of another short paper analysis by Raj Gupta.
• Multi-Agent Architectures: Instead of relying on a single ReAct agent, a team of specialized agents can improve performance.
• The "Inner Monologue" (IM)?framework from Huang et al. (2022b) uses language models for robotic action planning and decision-making and injects feedback from the environment.
• SayCan uses LLMs to predict possible actions a robot can take, which is then reranked by an affordance model grounded on the visual environments for final prediction.
• Selection-Inference divides the reasoning process into two steps of "selection" and "inference".
• STaR bootstraps the reasoning process by fine-tuning the model on correct rationales generated by the model itself.
• Faithful reasoning decomposes multi-step reasoning into three steps, each performed by a dedicated LM respectively.
• Scratchpad?finetunes an LM on intermediate computation steps and demonstrates improvement on multi-step computation problems.
• Least-to-most prompting for solving complicated tasks.

The Future of ReAct: Smarter, More Specialized Agents

To overcome these limitations, researchers are exploring new ways to organize AI agents:

• Multi-Agent Architectures: Instead of one agent doing everything, create a team of agents with specialized skills and responsibilities.
• Specialized Agents:?Creating agents specializing in specific domains (e.g., customer support and calendar scheduling) can improve performance and efficiency.

By carefully designing the architecture of AI agent systems, we can unlock ReAct's full potential and create knowledgeable, helpful, and reliable AI.

Conclusion

The ReAct framework represents a big step forward in AI. As AI agents become more sophisticated, they can help us with a wide range of tasks, from the mundane to the complex. By understanding how ReAct works, you can be better prepared for the exciting future of AI.

The ReAct framework was created to?synergize reasoning and action in language models?to solve diverse language reasoning and decision-making tasks.