Top Highlights from Microsoft Fabric Conference – Sweden 2024
Article by Dufrains David Mills Azure Data Architect
Over the past four days, the Microsoft Fabric Conference in Sweden has been packed with exciting announcements and developments that have captured the attention of the data community.??
David Mills, one of our Azure Data Architects at Dufrain has been closely following the updates, and he’s picked some key highlights that stood out.?
Here are our top 10 favourite updates from the event:?
1: Terraform Provider for Fabric
Terraform is a powerful Infrastructure as Code (IaC) tool that gives enhanced control and consistency across environments.? Integrating Terraform with Fabric allows users to streamline and automate complex Fabric deployments in various development stages (Dev, Test, Prod).?
This feels like a great advance for DevOps within Fabric!?
2: Access Databricks Unity Catalog tables from Fabric
In Fabric, it’s now possible to create a Mirrored Azure Databricks Catalog that creates a Fabric Shortcut to every table in the selected Databricks Catalog.?
This means no copying of data between Databricks and Fabric, and the Fabric items will be kept in synch whenever there is a change in the Databricks Catalog.???
This is awesome tech!?
3: Fabric Git Integration is in GA?
Fabric has been able to sync Workspaces to Azure DevOps for a while, but moving into General Availability is an important milestone to leverage proper version control of your Fabric components.?
4: Incremental Refresh for Dataflow Gen2
The ability to have incremental refreshes for Dataflow Gen2 has been long awaited.? This will reduce long-running job times as only new or updated data gets refreshed.?
This is a big win for minimising resource usage and returning your data faster!?
领英推荐
5: Invoke Remote Pipelines via Fabric ADF Pipelines
You can now call and run ADF or Synapse Pipelines inside of a Fabric ADF Pipeline!? This allows for more interactivity with the ETL on your Fabric and PaaS platforms.??
6: Database Migration Experience (Private Preview)
Migrating SQL Server or Synapse Dedicated SQL Pools to Fabric is complicated due to the differences between the SaaS and PaaS architecture.?
This tool promises to accelerate the process by automatically converting source? schema code to Fabric Data Warehouse format.? This feature is currently in Private Preview, but one we are watching closely as it could drastically reduce platform migration times.?
7: High Concurrency Mode for Pipeline Notebooks
This feature enables users to share Spark sessions across multiple notebooks within a pipeline.? When users trigger pipeline jobs, they are automatically placed into existing high concurrency sessions.?
This gives a much-improved notebook session start time and a huge notebook performance boost!?
8: T-SQL Notebooks
You can now use Fabric T-SQL Notebooks to execute your T-SQL code in the Fabric Warehouse.? This allows the developer to write code, document the process, and visualize the results in a single interface.?
9: Fabric SKU Calculator (Private Preview)
Estimating the correct Fabric SKU can be tricky to formulate depending on the client’s data estate and requirements.? Go too low and Fabric won’t perform well.? Go too high and the client will pay too much!?
The Fabric SKU Calculator could help hit the sweet spot by providing SKU recommendations based on data size, amount of data processed, plus other metrics.? It’s in Private Preview but looking forward to this going into Public Preview soon.?
10: DP-700 Fabric Data Engineer Associate Certification
This new exam is tailored towards Fabric Data Engineers in proving their subject matter expertise in data architectures, orchestration processes and data loading patterns.?
DP-700 is available to take from October 22nd 2024 in its beta phase.? We are expecting it to be as tough as DP-600 (Fabric Analytics Engineer Associate), but these Microsoft Certifications are well worth studying and passing.?
If you'd like to test out Microsoft Fabric, we also have a free PoC. You can find details here.