Marketing vs. Engineering: The Reality of Copilot Studio

As an Engineer of the Digital Workplace Collaboration team, I’m having the privilege to get intense hands-on experience implementing Microsoft user facing GenAI solutions in large enterprise environment. The unveiling of Copilot Studio at Microsoft Ignite 2023 was akin to the launch of a new spaceship, promising to take enterprise users to the unexplored realms of LLM. The much-anticipated native integration with SharePoint Online (SPO) was the rocket fuel for this journey. I mean, just add the SharePoint site link and you can speak to your SPO site, while MS takes care of the complexity

Well, reality hits hard: the initial launch was more of a test flight than a moon landing. The thing is, that until the last week, the parsing of SPO modern pages content simply didn’t work right. From the behavior and my experience in LLM I can assume that system was navigating the vast space of data using a keyword search, passing the top 3 results in LMA as context and summarization.

Last week’s Microsoft Build brought some upgrades to our spaceship. The latest version of Copilot Studio now includes an actually working generative AI capability. This has significantly improved the utility of Copilot Studio on SPO, enabling it to answer user queries with an acceptable level of quality that one would expect from GenAI based solution with GPT 4-turbo under the hood.

Despite these advancements, there are still a few very surprising asteroids in our path:

• SNOW Microsoft Graph connector: Currently, there are technical issues preventing the data source using the Microsoft Graph connector from passing Microsoft validation. It’s like trying to dock with the International Space Station using a paperclip and some duct tape.
• Session analytics: The current system is unable to parse raw sessions data using simple methods, resulting in incomplete reports and analytics in the portal. When we can’t see the sessions details, we are flying blind, and it is bad for continuous improvement of the bot performance
• Performance: the underlying model shows inconsistency and degradation during peak hours. It’s like our spaceship has a flat tire just when we need to hit warp speed.

These issues are currently limiting the performance and user experience of the product. It is my hope that Microsoft will address these issues promptly to fully realize the potential of Copilot Studio. The capabilities showcased at MS Build set a high bar, and I am optimistic about the future developments in this space.?And I will be more than happy to write an update to this article once the stuff shown to us on MS Build actually works.