So far, I had a mixed experience with the Copilot Studio. Sometimes it is amazing and sometimes it is like an un-finished solution in production.
- The auto-orchestration using GPT 4o is amazing, phrase-based orchestration is also good but GPT 4o is good, lot of time it saves and feels more like natural conversation with the system, phrase-based orchestration more feels like guided conversation. But I found sometime guided conversation is what you need to meet the requirements.
- The set of Topics what you get with the empty chatbot is very time saving, you can easily configure as per your requirement.
- The continues updates, recently GPT3.5 gets updated to GPT4.0 and there is a difference, the one which I liked most is the limitation of file size increased from 3MB to 500MB. I remember few years back when it was known as PVA, Power Virtual Agent, we rejected this platform to develop a food ordering system, we needed Adaptive Cards to display food items and other details in fancy way in Teams. But now there is lot of possibilities with different use cases. And for sure it will grow as any other Microsoft products grow from “good to have” product to “need to have” product.
- Ease of deployment, everything is ready for you, it’s a Low Code solution. You are just few click away to setup for different channels. You don’t need to setup things like spinning resource in Azure search, Azure OpenAI. No need of app registration, networking, and other peripheral technologies dependencies.
- Ease of development, or better to say ease of configuration. Also, with the help of Copilot you can create a ready to use sample bot for your problem statement.
- Excellent Dataverse integration, this is excellent, recently experience this and there is lot of configuration possibilities, and with the help of AI Prompts it exceeded my expectation.
- Actions – actions open a lot of opportunities to have different kind of data sources.
- The continues updates, yes, I listed it in positives, but at the same time it is a risk. I spent days to work around 3 MB file limit, but now my whole design is waste as limit is no more there with new GPT 4.0, but this is still fine, my bad experience is when bot started interpreting output of AI Prompts automatically and it started posting message two times, second one is the one which I put in place using Adaptive card, I had to delete logic what I put to next to AI Prompts. But story don’t end here, in a week those automatic responses stopped, just a days before go-live. It was one of the worst experiences I had.
- Unpredictability, it is something you cannot manage with GPT, no matter what your temperature setting is it will be unpredictable, so far, I never meet accuracy of 100 percent in any testing round. I guess this is something we need to live with in this AI era, success criteria changed from 100 percent to 90 or 80 percent.
- If there is issue raise a ticket with Microsoft. Sometimes it is annoying to not have control on the development environment. One example I can give you when bot suddenly stopped working in MS Teams, but still working perfectly in Copilot Studio test chat panel, as it is working fine in test chat panel, so the conversation map and debug feature are useless. But for other cases there is a better way, configure Application Insights, but to me it kills the experience of Low Code as soon as you go to Azure. May be blessing for developers but something out of scope for Citizen developers.
- There are few more minor issues, like bot is not fully compatible from channel to channel, for example citations didn’t work on Teams, but thanks to Microsoft recently I started seeing a popup in teams.
I have to say it is fun to play with, I found little unpredictability engaging, and uncharted territory of capabilities make is exciting.
It is a good to have product, if you have a use case where you can realize the ROI then why not? Like, if you are already invested in Microsoft O365 Infrastructure, or if you want to have GPT capabilities on your data in Dataverse then it is an excellent choice. For more control on the infrastructure and development better look for Azure Cognitive services.
I am still exploring it; Microsoft is continuously improving it; It may become promising for wider range of use cases.
