Microsoft Fabric with Dynamics 365 F&O for Demand planning
There is a lot of excitement around Microsoft Fabric and like many others, I wanted to explore its functionality with potential application in the D365 space. My curiosity for Fabric revolved around - what is it really, how can I use it to solve some business problems, where could it sit in the solution stack for D365 F&O and other dynamics apps, is it a tool for power users, business users, or developers, is it a cost viable solution?
I feel the best way to learn a new product is to try and use it to solve real problems. With that in mind I set some challenges to solve using Fabric
I was not successful in achieving all of my objectives for various reasons but here goes what I learned.
What is Microsoft Fabric?
First and foremost Fabric is a SAS solution from Microsoft. But it is different from other SAS solutions in that I would consider it a Server SAS not an end user SAS. Typically when you subscribe to say office, Power BI Pro or D365 Finance and Operations, you buy licenses. But with Fabric you don't buy licenses, rather you buy server size or capacity expressed in compute units.
There are a couple of ways of buying this server capacity, either through Power BI premium subscription model or you can provision an Azure resource (in your azure tenant) called Microsoft Fabric. And you connect this resource to your Fabric SAS subscription. This article explains the pricing and subscription model in quite a bit of detail - Microsoft Fabric Pricing Model - Everything you need to know! - Data Mozart - data-mozart.com.
Wanting to test Fabric out as an independent I went with the second route (create a fabric resource in Azure and connect it to the SAS solution). I first provisioned a Fabric resource in my Azure tenant.
The Fabric Azure resource can be provisioned from your azure portal and it comes in different sizes - F2 being the lowest tier to F2048 the largest size currently possible. A size essentially indicates the compute power/capacity that is available to you. As you can see below it can get really expensive and I am quite thankful to Microsoft for providing a sizing small enough for independents like me to try out. Surprisingly i found out that F2 works for most of my needs, the only time it was slow was when I was trying to query my data using code written in Spark notebook (which is probably not the best way to access the data anyway).
In many ways the functionality in Fabric is essentially a port of the Synapse functionality from Azure into a SAS solution. A smart strategic move from Microsoft that makes this product more accessible and usable to a wider audience/users. Having said that there is quite a bit of difference in the philosophy of the two products. You can read this article to get a good idea of the differences - Is Microsoft Fabric just a rebranding? - debruyn.dev
How do you access Fabric? - Fabric URL: https://app.fabric.microsoft.com.
How do you link your Fabric SAS solution with the Azure resource you created above?
Once you navigate to Fabric, navigate to any of your workspaces (you will have to figure this out a bit) and click on workspace settings and in the license capacity setting select your Azure resource.
Keep in mind that you can create new workspaces and each workspace essentially can connect to a different Fabric Azure resource. You also need to be assigned the role of Fabric Administrator in your office subscription to be able to manage this configuration.
Now we are ready to attempt solving our problems with Fabric. Let us look at each of your problems.
First some fundamentals I learnt about Fabric.
Workloads/application areas: In Fabric you currently have 7 workloads Data engineering, Data warehouse, Data factory, Data science, Real-time analytics, Data activator, Power BI. The following two resources give a good overview of these workloads - https://learn.microsoft.com/en-us/training/modules/introduction-end-analytics-use-microsoft-fabric/2-explore-analytics-fabric
Workspaces (Azure resource group?): Your first step should be to create a workspace. The workspace is a boundary within which you can create any fabric resource. A workspace to me seems to almost represent a resource group in Azure. Essentially you create a workspace, create all resources within the scope of a workspace and then finally if you are done then you can delete the workspace, which should delete all resources in its scope.
Data: The primary workloads you work with to manage your data seem to be Data engineering and Data Warehouse.
Lakehouse: To store data you can create a Lakehouse resource. To me a Lakehouse seems to be equivalent to a database (a very generic database) in which you can import files, create tables etc... Below is a screenshot of a couple of Lakehouse I created.
Lakehouse as a database?: When you click on your Lakehouse resource, you are presented with a Lakehouse explorer from which you can upload files, create table from loaded file, run SQL like queries to query data in tables and more.
Querying your data in Lakehouse: You can query the data in your Lakehouse using the SQL analytics endpoint highlighted in the figure above. You can create visual queries, which are pretty cool but you can also write straight up SQL queries.
OneLake: So what is my understanding of OneLake? Microsoft has indicated that there is only one OneLake per tenant. I think OneLake is essentially the underlying storage account(s). But as you saw above, you can create multiple workspaces and within each workspace you can create multiple Lakehouses. But the concept of a single lake would mean we can run cross Lakehouse queries or maybe even cross workspace queries. Below is an example of a cross Lakehouse query.
Essentially OneLake I think is the fundamental concept of unified storage where you can create all your data stores. The different services/functionality built in Fabric can leverage the shared OneLake.
Now on to solving our problems.
Retrieving and storing data: Establishing a connection and pulling data from D365 Finance and operations.
Transformation: Transforming the source data and exporting it to Azure data lake Gen2. This is so that the data can be imported by the new Demand Planning app from Microsoft.
Scheduling import and transformation: Create and schedule jobs within Fabric to pull data, transform it and send it to Gen2 storage account
领英推荐
Solution:
Needed Fabric Resources: Lakehouse, Data Pipeline, Notebook
Actions:
Pull data from the new Demand Planning app (in preview): Pull the generated forecast data from the new demand planning app from Microsoft
Solution: None so far. I have not been able to do this as there seems to be no easy way to access the data stored in the Demand planning app. The data in the demand planning app is stored in some backend Azure resource which I have not been able to figure out. Assuming I could figure out the tables / source of data in Demand Planning, I think a datapipeline would be the best way to pull data.
BPA application?: Business process automation will soon release a new update which will include connection to Fabric. This will allow us to bring the data sets from BPA into Fabric and let us customize/modify and merge with other data sources to generate custom reports.
Solution: Still waiting on BPA update to be released that will enable this functionality. Will test it out after.
Other functionality I am looking forward to explore is the Data science workload. I would like to build a custom model, tailored to one of our customers sales forecast data that can be leveraged by Demand planning app. Another area of focus is train a deep learning model. Both of these in the context of the F2 capacity and if that has sufficient horsepower to perform these activities in a timely manner.
Final observation - although Fabric is still primarily intended for technical users and developers, power users can certainly do a lot and making it a SAS application just made the application much more usable by less technical users.
Learning Resources and Reference Material
There are a number of resources I have used to bring up my knowledge on Fabric - Following is a listing of those blogs/posts.
Microsoft Learn: I would highly encourage everyone to go through the full learning course. It has detailed exercises that makes you notice the different areas of the Fabric workloads and helps you understand the functionality in a well structured manner - https://learn.microsoft.com/en-us/training/paths/get-started-fabric/.
Fabric Licensing and Pricing: This gives you a good idea on the different pricing options and the different ways you can subscribe to Fabric - Microsoft Fabric Pricing Model - Everything you need to know! - Data Mozart - data-mozart.com
What is Fabric? Is it Synapse?: If you have ever worked with Synapse directly in Azure, you probably have the same questions as me, how is Fabric different from Synapse? It seems quite similar. This post is a nice write-up highlighting how Fabric is quite different from Synapse while retaining the functionality - Is Microsoft Fabric just a rebranding? - debruyn.dev
Fabric Overview: This article provides a nice overview of the different workloads and the functionality available in Fabric - Thoughts about the Microsoft Fabric journey to becoming Generally Available - kevinrchant.com
Fabric - Lakehouse and Data warehouse: Once you start working with Fabric you start wondering about two workloads in Fabric - lakehouse and data warehouse. As they both deal with storing and managing data, when and where should you use each of these workloads? There is also recommendation on how to implement medallion architecture in Fabric. Fabric: Lakehouse or Data Warehouse? - debruyn.dev
Implement medallion lakehouse architecture in Microsoft Fabric - Microsoft Fabric - learn.microsoft.com
Shortcuts: Shortcuts are all the rage (in Fabric) and this a nice video that explains all the different shortcuts you can create - RULE 'EM ALL! Create OneLake shortcuts to ANY data source! - youtube.com
Semantic models in Fabric: I personally have not had a chance to explore this functionality in detail, but we need to get to know semantic models intimately as part of our Fabric learning journey - Semantic models in simple terms - blog.tabulareditor.com
Fabric Demo: Self-explanatory, it is a demo of Fabric functionality! - 220M+ row Microsoft Fabric demo using Direct Lake, Lakehouse, Warehouse, Spark and Pipelines - youtube.com
Blog with 25 articles on Fabric functionality: Great post(s) - there are about 25 articles that guide you through all the different areas of Fabric and providing you an overview and knowledge to jump start your exploration - Advent of 2023, Day 25 – Learning paths, materials, and documentation on Microsoft Fabric - tomaztsql.wordpress.com
Technical blog on Fabric nuances: This blog is highly technical in nature and digs into the underlying details of many of the fabric workloads/functionality - Lakehouse vs Data Warehouse vs Real-Time Analytics/KQL Database: Deep Dive into Use Cases, Differences, and Architecture Designs | Microsoft Fabric Blog | Microsoft Fabric - fabric.microsoft.com
Security: All of this learning is of no use without understanding how to manage the data security and user access in Fabric. This Microsoft Learn article is a great starting point - Microsoft Fabric security fundamentals - Microsoft Fabric - learn.microsoft.com