Advanced Analytics & Generative AI (end-to-end)

Imagine if you could easily build an end-to-end solution that allows you to query any type of data, whether structured or unstructured, in natural language. What if you could create useful dashboards and discover new insights effortlessly?

How integrating unstructured and structured data can provide comprehensive insights and enhance decision-making across various sectors?!

Some ideas / use cases.

Healthcare

• Patient Care Improvement: By analyzing unstructured data from patient records, doctors' notes, and medical images alongside structured data from electronic health records (EHRs), healthcare providers can identify patterns and predict patient outcomes. This can lead to personalized treatment plans and early intervention strategies.
• Drug Development: Pharmaceutical companies can combine clinical trial data (structured) with research papers and patents (unstructured) to accelerate drug discovery and development processes.

Finance

• Fraud Detection: Financial institutions can enhance fraud detection by integrating transaction data (structured) with unstructured data from emails, customer service interactions, and social media. This helps in identifying suspicious activities and preventing fraud.
• Customer Insights: Banks can analyze customer feedback from surveys and social media (unstructured) along with transaction histories (structured) to improve customer service and develop targeted marketing strategies.

Manufacturing

• Predictive Maintenance: Manufacturers can use sensor data from machines (structured) along with maintenance logs and technician notes (unstructured) to predict equipment failures and schedule timely maintenance.
• Quality Control: Combining production data (structured) with inspection reports and customer feedback (unstructured) can help in identifying quality issues and improving product standards.

Legal

• Case Management: Law firms can analyze case files, court documents, and legal precedents (unstructured) alongside client information and case outcomes (structured) to build stronger cases and improve legal strategies.
• Contract Analysis: By integrating contract data (structured) with negotiation emails and related documents (unstructured), legal teams can identify risks and ensure compliance.

But now, let’s dive in:

What if you want to enhance or automate the retrieval of relevant information and provide accurate, context-aware responses by using GPT technology? Uploading or just automating by sending attachments to a team mailbox?

What if you use a hybrid approach with OCR and Large Language Model to get better results without any pre-training?

In this Solution we use Azure Document Intelligence combined with GPT4 vision for unstructured data. With Cosmos DB, the extracted information from the documents is stored and mirroring into Microsoft OneLake allows access the data in near real-time. Some more details around mirroring in Microsoft Fabric can be found here: Mirroring - Microsoft Fabric | Microsoft Learn

From there, Advanced Analytics, BI and AI solutions using Microsoft Fabric, we consume all the information over a dashboard / report. For the document intelligent solution, you can use this GitHub repo and follow the steps.

Document extraction and quick check on the extracted data:

In Microsoft Fabric, you create first a Workspace and a Lakehouse. For more guidance, follow the steps provided here: Lakehouse end-to-end scenario: overview and architecture - Microsoft Fabric | Microsoft Learn

In this example I'm using a public dataset provided from Kaggle from the insurance industry where the goal is Predicting the likelihood of claims for risk assessment and policy pricing. Insurance Claims Dataset (kaggle.com)

As a data replication solution, Mirroring in Fabric is a low-cost and low-latency solution. In this case I have mirrored Cosmos DB and you can see the same information in Microsoft Fabric.

I have created 2 Semantic models. One from the Claims dataset and the other one from Cosmos DB where I have the output of the processed documents.

The Policy subscription distribution was created with the help of Power BI copilot and the other Chart is now representing the documents which have been processed. This example shows how many pages per document. Of course, you can adapt/change and add more metrics or details.

And recently announced as public preview, AI skills allow you to create your own conversational Q&A systems on Fabric using generative AI. You can give your colleagues the experience of simply asking a question and getting a reliable, data-driven answer in return. Here I'm asking, "What age owns more policies?"

And shortly cross-checking with the dashboard, looks like it’s 100% right.

The flow of data from the data source to its destination is key! With the Lineage Capabilities in Microsoft Fabric, you have everything under control.

To effectively secure and govern your data, manage various data types (structured, unstructured, AI-generated) across multiple clouds, and stay updated on current and future regulations is where Microsoft Purview simplifies to ensure a comprehensive and resilient data governance framework.

Some Links

• ARGUS: GitHub - Azure-Samples/ARGUS
• Microsoft Purview: https://www.microsoft.com/en-us/security/blog/2023/12/07/new-microsoft-purview-features-use-ai-to-help-secure-and-govern-all-your-data/
• Lakehouse Microsoft Fabric: What is a lakehouse? - Microsoft Fabric | Microsoft Learn
• AI Skill: AI skill scenario (preview) - Microsoft Fabric | Microsoft Learn
• Governance: Lineage in Fabric - Microsoft Fabric | Microsoft Learn

Here the end-to-end video.