Boosting AI Performance: How a Two-Agent System Outshines OpenAI's Latest Model

The latest OpenAI model o1-preview-2024-09-12 just became available. This model aims to improve problem-solving as it has been designed to spend more time thinking before responding. It can solve more complex science, coding, and math problems better according to OpenAI.

Since I have been working with multi-agent frameworks that similarly improve the LLM response quality, I wanted to test this new o1 Preview model against my multi-agent framework that utilizes GPT4o.

Both the o1 Preview and my two agent framework received the same prompt: "What is Claro Analytics?". This article shows how you can add reflection to the multi-agent workflow and get a noticeably better-quality result than using the o1 Preview alone.

The Two-Agent Setup: Breaking Down the Task

I created two specialized agents. Each one has a unique job to help solve complex queries more thoughtfully. The reason why I like customized AI frameworks is that you can easily tailor agents for various roles, divide tasks, and use reflection and collaboration to get superior results.

Agent 1: Decomposer/Planner

The first agent's job is to take a big, complicated question and break it down into smaller, manageable parts. The user can monitor what the agent does exactly as the process goes on. This is a huge benefit as we can audit the agent's thought process.

Agent 2: Solver/Refiner

Once Agent 1 has finished breaking down the question, Agent 2 steps in to solve each part. After that, it reviews the entire solution to make sure it is cohesive and clear. As a user, you can adjust the agent's goal and backstory to suit your needs. You can also provide the agents with tools you think will be needed for a given task.

Reflection Loop: Getting It Just Right

This framework uses a reflection loop. Once Agent 2 generates the response, it reflects on it, making sure everything is accurate and well-organized. By going through this reflective process, the answer ends up being higher quality and reliable.

How It Outperformed o1

To test this setup, I asked both the two-agent system and the o1 Preview the same question: "What is Claro Analytics?". Just for reference I also asked this question from the GPT4o model and its response was rather short and basic. The response time for GPT4o for this question was 2.85 seconds.

The o1 Preview’s response was much more structured and fairly good compared to GPT4o. However, the quality wasn’t flawless, as there was a clear data error regarding the company that had acquired Claro Analytics. Getting erroneous information from an LLM is a major red flag. The response time for the o1 Preview was 18.9 seconds. Inference time is noticeably longer compared to GPT4o but the quality just wasn't there, unfortunately.

My two-agent system gave the best and most impressive answer. By breaking the task down with Agent 1 and reflecting on the response with Agent 2, the system provided a more coherent and accurate answer, without any errors. It even included information about when the company was founded and also mentioned the founder Michael Beygelman in its response. The response was extensive and better categorized. It offered info on how the platform can be used in different scenarios and included information about the competitive landscape. My model took the longest time to generate a response: 43.5 seconds. However, since this was about reflection and being able to give better answers, speed isn't our primary concern. I will always choose a better answer over a faster inference time.

Conclusion

Using a multi-agent system improves the quality of the AI model's responses by adding a layer of reflection and refinement that can be customized for your needs. Also, it will allow you to use a plethora of other LLMs that might be more cost-effective yet produce equal or better results. In this case, the result of my multi-agent framework for this specific question was better than what I got from the o1 Preview.