Microsoft Fabric’s Oct-Nov 2023 released features

Microsoft Fabric is nearing its General Availability, and the Microsoft engineering team is working non-stop to add new features. Here is a summary of the main updates and enhancements made to Fabric between October 2023 and November 2023 so far. These new features are designed to improve the platform's performance, security, and user-friendliness:

?

1.????? Column-Level & Row-Level Security for Fabric Warehouse & SQL Endpoint: Microsoft announced the public preview of Column-Level and Row-Level Security in Fabric Warehouse & SQL Endpoint. Column-Level Security allows restricting column access within the Warehouse, thereby enhancing data security by enforcing access restrictions system-wide. Row-Level Security, on the other hand, controls access to rows in a table based on user accounts, with rules enforced database-wide to minimize security vulnerabilities. These features simplify the design and coding of security and are managed within the Warehouse or SQL Endpoint table, enforcing rules whenever data is accessed.

2.????? Microsoft OneLake Shortcut Support to Power Platform and Dynamics 365: Microsoft OneLake now supports creating shortcuts to Dynamics 365 and Power Platform data within Dataverse, allowing data analysis with Microsoft Fabric without needing to export data or build ETL pipelines. OneLake acts as a single unified data lake, and with this update, it virtualizes data through shortcuts that appear as if they were physically present in OneLake. This integrates the data with the lakehouse, formatted using the Delta/Parquet standard, streamlining data analysis with Spark, SQL, and Power BI reports without data transformation.

3.????? Data Activator Public Preview: Data Activator is now available in public preview for all Microsoft Fabric users. This feature enables automatic alerts and actions from Fabric data, eliminating the need for manual monitoring of operational dashboards. Data Activator allows users to create alerts directly from Power BI reports and is designed to be user-friendly, requiring no technical knowledge. It can trigger alerts and actions such as sending emails, Microsoft Teams messages, and trigger Power Automate flows for various conditions set by the user, such as monitoring overdue invoices or checking shipment statuses.

4.????? Fabric Roadmap: Responding to feedback for more transparency, Microsoft has announced the Fabric Roadmap, which will be regularly published to provide insights into upcoming features and their expected release dates. This roadmap will eventually be merged with the Power BI roadmap from the Power Platform release plans, offering a unified view of upcoming developments for Microsoft Fabric.

5.????? Semantic Link for BI and Data Science: The Public Preview of semantic link has been introduced, connecting Power BI datasets with Synapse Data Science within Microsoft Fabric. This enables access to Power BI data for notebooks and Python, supporting pandas and Spark APIs, and allowing the computation of Power BI measures, reading tables, and executing DAX queries. Semantic link facilitates data augmentation, validation, exploration, and an extendable set of semantic functions for a comprehensive data science environment.

6.????? Semantic Kernel in Microsoft Fabric: Semantic Kernel is introduced as part of Microsoft Fabric’s Lakehouse, offering an LLM-based application framework. It enhances the connection between Prompt and Code, supporting multiple programming languages and enabling seamless Copilot applications. The framework allows for managing both structured and unstructured data, utilizing traditional techniques like Python’s libraries and T-SQL for data extraction and analysis. It also provides the ability to chat with enterprise data, creating interactive data management and analysis scenarios.

7.????? Enhanced UX on Event Processor and Expanded Custom App Connections: Microsoft Fabric's Eventstream has undergone updates aimed at improving efficiency and user experience. The Event Processor now features a no-code editor that simplifies the design of event processing logic, allowing for real-time processing with powerful temporal functions. The recent UX improvements facilitate adding or deleting operations with intuitive interactions. Additionally, the Custom App connection strings have been expanded to support protocols such as Kafka and AMQP, enabling seamless connection to Eventstream for applications and Kafka clients. This update emphasizes flexibility and ease of use for real-time event data ingestion and processing within Fabric.

8.????? Fabric Warehouse publishing full DML to Delta Lake Logs: The Data Warehouse in Microsoft Fabric now automatically publishes all Data Manipulation Language (DML) actions (Inserts, Updates, Deletes) for each table to their Delta Lake Log in OneLake. This is part of a broader vision to eliminate data silos and facilitate data sharing across teams and services without the need to duplicate data. By leveraging Delta Lake as a common format, data remains secure, easier to manage, consistent across reports, and readily shareable. These enhancements ensure that changes made in Data Warehouse tables are reflected in the Delta Lake Log within a minute, simplifying data integration for various uses, including Machine Learning and AI model training.

9.????? Data Warehouse SKU Guardrails for Burstable Capacity: Microsoft Fabric has introduced SKU guardrails for burstable capacity in its Data Warehouse and SQL Endpoint. Burstable capacity allows workloads to temporarily use additional resources to enhance performance, particularly useful for uneven workload patterns. The feature implements 'smoothing' to average out over-usage over time, thus optimizing capacity management. The new SKU guardrails ensure that this bursting ability remains within reasonable limits, preventing overconsumption of resources. With guardrails, overconsumption can be managed more effectively, maintaining performance without the risk of rapid resource depletion. This feature is automatically active for new tables created within Data Warehouses on Microsoft Fabric.

10.? Time-Travelling through data: Microsoft Fabric has introduced a feature allowing users to clone data warehouse tables, capturing a snapshot of data as it existed at a specific moment in time(without data duplication), up to a seven-day retention period. This enables historical trend analysis, aids in compliance requirements, and assists in root cause analysis during data discrepancies. The cloning process is user-friendly, allowing multiple tables to be cloned at once through an intuitive interface or T-SQL, streamlining reporting, analytics, development, and testing processes.

11.? Fabric Warehouse user experience updates: Fabric Warehouse rolled out several user experience enhancements designed to increase productivity and streamline user interaction. These include a new UI for cloning tables, saving queries as views or tables via the Visual query editor, an endorsement feature to highlight quality data assets, enhancements to the sample loading experience for better performance, and enabling viewers/shared recipients to save queries.

12.? Automatic Data Compaction for Fabric Warehouse: Automatic data compaction has been announced for Fabric Warehouse, aiming to automate maintenance tasks to improve performance and reduce costs. The feature rewrites multiple smaller Parquet files into fewer larger ones, enhancing read performance and removing deleted rows from files. This compaction occurs as system tasks generated by the engine, working behind the scenes without additional effort from the user.

13.? Create multiple shortcuts faster and easier: Microsoft OneLake, as a unified storage location, now allows users to create multiple shortcuts in a more straightforward manner. Users can select multiple targets at once and create new shortcuts in a single operation, reviewing and editing selections before creation, which significantly improves the efficiency of data management and accessibility.

14.? Delta-RS introduces native support for OneLake: Delta-RS, a native Rust library for Delta Lake, has extended support to OneLake. This improvement enables direct management of lakehouses in OneLake using Delta-RS, facilitating integration with ADLS Gen2 compatible tools and libraries. The OneLake team has collaborated with the Delta-RS community to ensure seamless functionality and compatibility, promoting the integration of a broader range of data management tools.

15.? Improvements to CSV data ingestion in Fabric Warehouse: The introduction of CSV file parser version 2.0 for COPY INTO commands in the Data Warehouse has been announced. Based on Microsoft Research's innovation, the new parser significantly improves CSV file ingestion performance, with tests revealing a 38% improvement in ingestion times on average, and in some cases, speeds up to four times faster than the legacy parser. The new parser has been made the default option, simplifying the process for users and providing immediate performance benefits.

16.? A New and Improved Experience for real time analytics data ingestion: Microsoft Fabric has rolled out an update that simplifies the data ingestion process for KQL databases. This is achieved through a new user interface called the Get Data experience, which reduces the number of steps and clicks needed to import data. It includes a Guided Wizard that assists users step-by-step, ensuring an intuitive process whether users are working with local files, Azure storage, Amazon S3, Event Hubs, Eventstream, or OneLake. Real-Time Analytics now supports these six different data sources, allowing for efficient data import into KQL databases with minimal effort.

17.? Fabric Warehouse Support for sp_rename: Microsoft Fabric's Data Warehouse now includes support for sp_rename, a feature that allows users to rename user objects like Tables, Stored Procedures, Functions, etc. The update ensures that when objects are renamed, their new names are reflected in OneLake within one minute, which helps maintain consistency and streamlines the data management process.

18.? Dynamic Data Masking for Fabric Warehouse and Lakehouse SQL Endpoint: The introduction of Dynamic Data Masking (DDM) for Fabric Warehouse and SQL Endpoint in Lakehouse is a pivotal update for data security. DDM enables the protection of sensitive data by masking it according to user-defined rules without changing the core data structure or application logic. It provides enhanced data security by concealing sensitive information from unauthorized access, seamlessly integrates with existing applications without requiring changes, and offers fine-grained control over the masking rules.

19.? Using Your Own Library with Microsoft Fabric: Microsoft Fabric has provided a comprehensive guide for creating and using personal libraries within the Lakehouse environment. This feature allows for the reuse of pre-compiled Python code, such as business logic functions, across the organization.

?

References:

?

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-column-level-row-level-security-for-fabric-warehouse-sql-endpoint?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/microsoft-onelake-adds-shortcut-support-to-power-platform-and-dynamics-365?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-the-data-activator-public-preview?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-the-fabric-roadmap?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/semantic-link-use-fabric-notebooks-and-power-bi-datasets-for-machine-learning-data-validation-and-more?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/chat-your-data-in-microsoft-fabric-with-semantic-kernel?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/whats-new-in-eventstream-enhanced-ux-on-event-processor-and-expanded-custom-app-connections?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-publishing-full-dml-to-delta-lake-logs?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/data-warehouse-sku-guardrails-for-burstable-capacity?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/time-travelling-through-data-the-magic-of-table-clones?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/fabric-warehouse-user-experience-updates?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-automatic-data-compaction-for-fabric-warehouse?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/create-multiple-shortcuts-faster-and-easier?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/delta-rs-introduces-native-support-for-onelake?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-improvements-to-csv-data-ingestion-in-fabric-warehouse?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/get-data-in-real-time-analytics-a-new-and-improved-experience?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-fabric-warehouse-support-for-sp_rename?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/announcing-dynamic-data-masking-for-fabric-warehouse-and-lakehouse-sql-endpoint?ft=All

·?????? https://blog.fabric.microsoft.com/en-us/blog/fabric-changing-the-game-using-your-own-library-with-microsoft-fabric?ft=All