How Do I Improve My AI Builder Model?

Once you have completed you AI Builder model, you release it in to production, and that is the end of it! Yay! BUZZZZZZZZZZZZZZZZ!!! True to the notion that no software ever survives contact with the end user, about an hour after launch you will get a call saying that some information is wrong and the model isn't working as expected.

So... is there a way to tag and retrain the model with more real world data? Yes, actually there is!

You want to implement something called a Feedback Loop (this feature is currently in PREVIEW at the time of the authoring of this article). You set a threshold limit on one or more fields that you want to improve on. The feedback loop will will make note of the times when your model exceeds your threshold, and make that information available to you when you have enough data to retrain your model.

So, say you have a model on a Contract document. You want to make sure that your model will continually improve with the contract value. You want to set a limit on the confidence that anything under 70% gets written to the feedback loop.

In your flow you can set a condition that looks at the confidence score, and then calls the Save file to AI Builder Feedback Loop action. The action will save the file, model name and model output to a special Dataverse table within the same environment as your AI model.

NOTE: One thing you DO need to be aware of is that the ENTIRE document will be saved to this Dataverse table. That will eat up your capacity if you are not careful!!!

Once you have some data here, you can then go in to the Add documents area of the model and select Feedback loop as a data source. Tag the documents and retrain your model!