Microsoft or Amazon? Your Key to Building Intelligent Chatbots

Spent couple of hours playing with Amazon Q Business and Microsoft copilot studio. Here are a few of my observations:

Both products are easy-to-use, with user-friendly UI. You can navigate either portal seamlessly and have a bot running within minutes. However, there’s a catch. It’s like owning a beautiful car with a great interior, advanced electronics, and a stylish trim, but one that only starts occasionally. Most of the time, it just won’t run.

Remember those black-and-white videos of the first cars, where people had to crank the engine multiple times to start it? Out of five attempts, the car might run once, only to stop after a few miles. Building a chatbot today feels much the same.

We are in the Model-T era of chatbots.

Despite the sleek interfaces and promising features, the functionality is often unreliable, making the process frustratingly inconsistent.

Remember, this technology lives in the cloud, so it’s continuously improving. The journey from the Model T to Tesla took more than 100 years, but we can go from Model T chatbot to Tesla chatbot in the next 1-2 years. When given the Math benchmark, a set of complex problems used to assess the problem-solving ability among the best young mathematical talent in the United States, GPT-3 went from getting 5% correct in 2021 to GPT-4 getting 90% correct in 2024. While there are shortcomings today, these can improve rapidly. I have a great deal of faith in the exponential growth and maturity of AI.

Even my today's test keeps AI below the trendlines, but I firmly believe Leopold Aschenbrenner 's claim in his article situational-awareness that - it is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer.

Microsoft gives a way to customize the Copilot, it is a nice escape hatch, it is like hard code few things or insert some logic for a better customer experience. I think, that is very smart thing to do. If you see AI is failing in some instances, you can customize the Copilot for particular prompts.

Microsoft Copilot’s ‘actions’ feature is a game-changer, enabling seamless interaction with external data and tools. From fetching real-time information (like weather or stock prices) to manipulating Excel files, running scripts, or even automating desktop flows, the extensibility is remarkable. This opens up a world of possibilities for multi-agent collaboration, making complex problem-solving achievable.

Microsoft has laid the groundwork for a thriving ecosystem, with all the tools in place for sophisticated multi-agent orchestration. It’s a truly impressive feat and reminds me Jeffrey Snover comment (paraphrased) that Microsoft fails to consistently make mistakes. They repeatedly bounce back and emerge stronger. If you’re wondering who will dominate the enterprise sector, look no further than Microsoft’s extensibility story with Copilot. Their ability to innovate and improve continuously is a testament to their future leadership.

By contrast, Amazon Q Business does not allow this or any level of customization. I appreciate the AWS approach to building applications using prompts. It’s incredibly useful for creating documents with snippets from various sources, which can be beneficial in many business scenarios. However, unlike Microsoft Copilot, Amazon Q Business lacks customization options. It’s time for their product managers to take a cue from Microsoft’s playbook and consider adding this feature.

Both AI-powered Chatbots from Amazon and Microsoft have shown promise, their current implementations often fall short in accuracy and quality. In numerous tests, simple keyword searches using a Unix grep command could have yielded more relevant results compared to the responses from these multi-million-dollar AI systems.

This highlights the need for significant improvements in model performance, particularly in understanding context, interpreting complex queries, and accurately extracting information from documents. It’s crucial for these systems to evolve beyond simple keyword matching and develop a deeper understanding of natural language and meaning.

Between Amazon and Microsoft, my experience reveals distinct differences in their performance. When tested with identical documents, Amazon Q Business consistently generated more comprehensive and detailed answers to the same questions as compared to Microsoft Copilot. Copilot’s responses often remained succinct, limited to 2-3 lines, even with preview features enabled. This suggests that Microsoft’s models may prioritize brevity over depth, potentially overlooking important nuances in the training documents.

Conversely, Amazon Q Business demonstrated a stronger ability to extract and synthesize information, resulting in more elaborate and informative answers. This indicates that Amazon’s models may be better equipped to handle complex document structures and provide deeper insights.

It’s important to note that this is a limited sample size, and further testing across diverse document types and domains is needed to draw definitive conclusions. Nonetheless, my initial comparison suggests that Amazon Q Business may currently be a more suitable option for users seeking in-depth document analysis.

There are not many knobs behind the scenes to tweak and improve them. I believe because this black box is built over the black box of LLM. So, the developers are getting away from real technology by 2 black boxes. There can be consequences for developer down the road.

Both Amazon Q Business and Microsoft Copilot fell short when I asked them to identify me based on my LinkedIn, X.com (formerly Twitter), and blog links etc. Instead of offering insightful analysis, they simply regurgitated snippets from one of my blogs. It’s disheartening to see $200 million AI platforms underperform compared to online scammers who can effortlessly map your entire social network using outdated technology.

Ease of Use

Both platforms are easy to use, as long as you stay within their default settings. If you deviate from these defaults, you’ll face the complexities of configuring IAM, permissions, policies, and other security settings, which will unleash the wrath of AWS security upon you.

Both platforms offer a plethora of connectors, allowing you to connect to almost any kind of data source. However, I am unsure how well these connectors will perform, given the current level of intelligence of the LLMs.

For this analysis, I specifically focused on Amazon and Microsoft’s offerings, even though similar tests on OpenAI yielded superior results. OpenAI’s limitation on saving GPTs beyond a few documents prevented a direct comparison. Despite this constraint, OpenAI with limited documents still outperformed its competitors. This raises an intriguing question: is Microsoft truly utilizing OpenAI technology behind Copilot Studio? Evidence suggests they may be developing their own LLM, potentially using Office customers as unwitting beta testers.

There are so many differences between Amazon Q Business and Microsoft CoPilot Studio, that makes me wonder if this even an apple-to-apple comparison. Microsoft is way ahead in their integration and customization story.

Although chatbots created by simply dropping documents and clicking a few buttons can effectively handle basic questions, they are well-suited to replace Tier 1 customer support personnel for addressing basic, repetitive queries.        

In conclusion, while these technologies are still in their early stages, with varying degrees of aptitude, we should embrace experimentation and gradual implementation of them. Familiarize yourself with their capabilities, provide feedback to the developers, and remember the pace of this technology is exponential. In just the last two years, GPT-4 has amazed us with its capabilities. It can solve complex math problems and communicate like a human, exhibiting emotions and empathy—a truly remarkable technological achievement.

I will leave you with these test scores of AI system.

#AI #Microsoft #Amazon