Evaluated the latest Google Vertex AI Agent product, and the results are quite interesting!
- Not Quite "Agents": First of all, it's definitely not "agents." It responds to the OpenAI Assistants, but with a nicer UI. However, it only works with the pretty bad Gemini 1.0 model at the moment.
- Gemini 1.5 Prospects: Not sure that Gemini 1.5 will help here. It is slow and expensive. On the other hand, this product is not designed to be smart, and may not require a lot of horsepower, focusing instead on easy control of bot behavior and integration with the Google ecosystem.
- Tools and Assistants: You can add tools, similar to OpenAI Assistants - OpenAPI specs, JSON schemas, etc. The only built-in assistant is a code interpreter and it is very bad. And yes, Google's product does not have a Google search tool.
- Lack of Self-Reflection and Error Correction: AI agents intended to self-reflect, and to some degree, can evaluate own results and correct themselves. The code interpreter can work ONLY if it can correct its own mistakes. ChatGPT excels at this, it's amazing. The Google version just fails on the first error. I was not able to make it work.
- Missleading "Agent" term: The word "agent" feels like a sneaky play on words, from the customer support area with real human agents. And overall this product first of all focuses on customer support cases. So I can understand why they picked this word for this use-case, but it is not "AI agents".
- Agent-to-Agent Communication: You can make an agent call another agent in the prompt, but it's just message redirection. No real agent-to-agent chit-chat. It's like redirecting to another AI to answer, that's it. It somehow mimics existing "human" support channels. I guess it's easier to sell and explain such…
- Customer Support and Integration: This feels like ChatGPT for your website or basic customer support. You can provide examples, build some logic flows, and even tell it when to escalate to a human by setting the message state. Integration-wise, it plays nice with FB Messenger, Google Chat, Slack, Telegram, SMS,…
- Datastores and Crawling: Types of data stores you can add are impressive. You can crawl websites, add SQL databases or upload files, etc. However, I have not tested its quality. I tried to add some website to crawl, but it said that I need to validate domain ownership, so very limited.
- Overall Impression: So, while I appreciate Google's initiative here, and understand that the main audience here are enterprises looking to simplify customer support, using the word "Agent" is misleading and there is no revolution here. We had such products for a year already, just not from the big players.
- Marketing vs. Reality: On presentation, it sounds like a revolution in AI, changing the ways how companies work etc. In fact, we got a boring product for enterprises. Which is still ok, and it will make a bunch of money, but this functionality you would expect from any big cloud provider these days.
- Product Ecosystem and Naming: And as usual with Google, it is very confusing to see a lot of various meaningless acronyms for the products, Vertex, Dialogflow CX, Dialogflow ES, some other terms I do not remember which already got deprecated, and they try to reach feature parity with a new, while keeping…
- Git Integration: From the cool things, there is a Git integration. I have tried it out, and you can have a decent CI/CD flow with it. It stores all the prompts, configs, tooling data, etc. I would love it to become standard for other products.
