Test Driving OpenAI Operator Agent

As part of my ongoing exploration of AI, I recently downloaded the Pro version of ChatGPT to evaluate its performance and compare it with our in-house LLM. Since Operator isn't available in the Netherlands yet, I had to use a VPN to connect via San Francisco - from our LA Office in Irvine.

Once in, I gained access to the Operator Agent, a new feature designed for complex, interactive tasks. Naturally, I decided to push its boundaries with an ambitious travel-planning challenge.

TL;DR? See the Video Yourself (It is 5x speed since it took me a long time ??)

So Why Test Operator?

1. Comparison with my own LLM & Agentic Design Strategy I wanted to see how ChatGPT’s Operator Agent stacks up against our in-house large language model for real-world tasks.
2. Complex Travel Use Case Rather than limiting the test to simple tasks like ordering food, I planned an intricate, multi-country trip to the Middle East—including Oman, UAE, Saudi Arabia, and Egypt.
3. Hands-On Evaluation By booking flights, hotels, SUVs, and limos, I aimed to push the Operator Agent’s boundaries in navigating websites and filling out forms.

The Task: A Three-Week Middle East Tour

My itinerary was ambitious:

• Fly Amsterdam → Muscat (Oman)
• Rent an SUV for a week of exploration in Oman
• Continue onward to UAE, Saudi Arabia, and Egypt
• Book flights or rentals between each destination
• End the trip with a flight back to Amsterdam

In principle, it’s not that different from booking flights and hotels individually—just repeated multiple times, with some extra details like specifying an SUV in each country.

What Worked

The Operator Agent demonstrated its potential for handling sequential tasks. It carefully attempted to navigate the virtual environment to:

• Search for flights, SUVs, and hotels
• Follow logical steps, such as asking for personal details and refining options
• Execute subtasks in a relatively structured way

For a feature still in its experimental stage, it was promising. The concept of managing these tasks via a conversational agent could save significant time when perfected.

What Didn’t Work

While the concept is ambitious, the execution left much room for improvement:

1. Slow Performance The agent struggled with responsiveness. Even with my strong internet connection (500 Mbps via VPN), the latency was significant. This suggests resource constraints on OpenAI’s end, potentially due to limited compute capacity or the high complexity of recursive task execution.
2. Interface Limitations The Operator’s interface mimicked a virtual machine, which felt clunky and blurry. Navigating through drop-down menus (e.g., selecting nationality) was painfully slow, sometimes requiring manual intervention. This added friction to the experience.
3. Overhead and Complexity The multi-agent system seemed computationally heavy. Every action felt like it involved several layers of back-and-forth processes between agents, making the system seem overwhelmed. For something as basic as booking a flight and rental car, this complexity felt unnecessary.
4. User Experience Challenges Despite its logical approach, the agent often behaved like a novice travel agent, asking too many clarifying questions and taking too long to process straightforward tasks. This “human-like” interaction model might be better suited for simple tasks but falls short in handling large, nuanced operations efficiently.

Agentic AI Has a Loooong Way To Go (for complex multi-task tasks)

While the Operator Agent shows potential for personal use cases like travel or meal planning, its true test will be in enterprise applications. Tasks like document analysis, contract generation, and knowledge-base searches require not only speed and accuracy but also seamless integration into existing workflows. Currently, the system feels far from ready to tackle such scenarios effectively.

What OpenAI needs to improve

1. Speed Optimization: Address latency issues and ensure the system can handle tasks in real-time.
2. Enhanced Interface: Replace the virtual machine-style environment with a smoother, more intuitive UI.
3. Task Efficiency: Simplify task execution to reduce unnecessary back-and-forth between agents.
4. Scalability: Build infrastructure that can handle millions of simultaneous users without degrading performance.

Final thoughts

The Operator Agent, while a groundbreaking concept, remains experimental. For simple tasks like booking restaurants or flights, it might work reasonably well with further optimization. However, for more complex or enterprise-level tasks, it has a long way to go.

That said, 2025 is shaping up to be the year where AI redefines many aspects of how we live and work. As companies like OpenAI continue to iterate and improve, we may yet see a future where such tools become indispensable. Until then, my travel plans are still safer in my own hands—at least for now.

Stay tuned for more updates as I continue testing this and other AI systems!