Towards a fully automated document intake with Power Platform

The rise of Generative Artifical Intelligence and subsequent introduction of Large Language Models into Microsoft Power Platform, bring with them a tremendous set of features for intelligent processing of data and documents in a Low code fashion. Microsoft Power Platform has been an AI infused Low Code solution for quite some time already. The AI Builder component provides specialist pre-trained machine learning models for a lot of popular tasks, like document extraction, object detection and more.

With AI prompts Microsoft has further expanded the AI portfolio of the platform into a more general direction. AI prompts allow to use natural language to create instructions handled by the underlying GPT-based LLM. The out of the box GPT model is constantly kept up to date with new versions as they become available, thus enabling users to automatically get the latest and greatest from the worlds most cutting edge LLM.

Each of the those components - AI Builder and AI prompts - serve their purpose. AI Builder provides specialized functions for specific tasks with a deterministic output, while AI prompts deliver the freedom of Generative AI and natural language to tackle any task in a more creative way. Combining these two methods of artificial intelligence leads to a whole new way of formulating solutions once deemed very complex to model.

Lets take a look at a popular sample from the Hyperautomation area:

Document Intake Automation

Figure 1 shows the process that the solution enables. Most parts of the solution are available from the GitHub repository , with the exception of the pre-trained AI builder models.

Starting at the intake phase, we can easily listen to an intake of choice (DMS library like SharePoint, Confluence, File Share,...) by modelling a Power Automate cloud flow. See here for a complete guide and list of available integrations . For the sample, the intake will be a shared mailbox, where emails and attached documents are received.

The highlighted flow in figure 2 is the main intake orchestrator and has two tasks.

• Listen for new emails to arrive
• Classify the content based on the content of the email using AI prompts
• Save the attachments (which are the documents themselves) in the correct location for subsequent extraction by specialized AI builder models

Figure 3 shows the first step - Classify and approve text. The flow applies an AI prompt to the email body and subsequently notifies a human of the results as part of a "human in the loop" approach. This manual approval could be ommited after a certain "stabilization" phase.

Figure 4 shows the AI prompts library in Power Platform. AI prompts are modeled in a Power Platform environment and are available (under respective security roles) to any Flow, Power App or Copilot. They can be thought of as a library of business functions expressed in natural language - a very powerful concept!

Figure 5 shows how the prompt used by the intake flow is designed. AI prompts are a combination of natural language instructions and placeholders that can contain inputs. In Figure 5 a sample text that represents a dunning letter email is used to test the prompt. The AI prompt builder allows rapid, iterative development of these prompts by enabling direct testing in the designer, as can also be seen in the figures 5 and 6.

With the email now correctly categorized, the Flow continues to store the attached documents (which we assume relate to the content in the email) in the correct location, from where they will be picked up later by the specialized flows using the specialized AI Builder models tuned to extract information from each specific type of document.

For the sample a new order delivery note was classified from the email, now a second flow applies an AI Builder document extraction model to the documents.

Figure 8 shows the flow that extracts the information from the document. It uses a pre-trained model to identify the information and store it in Dataverse. Figure 9 shows the AI Builder model trained with sample delivery notes.

Figure 10 shows the generated and extracted data in Dataverse. A simple Model Driven Power App can be used to work with the data.

Going 1 step back into the flow that does the data extraction, we can also see the Flow using any of its 1200+ connectors to Line of business systems to act on the freshly extracted data. In Figure 11 we can see how an SAP connector is used to fetch product information extracted from the document from an SAP instance.

Conclusion

This sample shows how effectively and easily Power Platform allows to model once complex processes using Generative AI and traditional Machine learning techniques and its own set of functions. Make sure to head over the Github repository to get the sample solution (provided as-is). NOTE: The sample solution does not contain the AI builder models as they have been trained on confidential data.

References

GitHub Repository: PowerPlatform/AI/AADocumentIntelligencePublic at master · aschauera/PowerPlatform ( github.com )