Introducing Mirroring: The Future of Data Ingestion with Microsoft Fabric & Cognizant

By: Channa Wijesinghe

Are you tired of writing incremental load logic or identifying watermark columns for data ingestion? The process can be time-consuming and repetitive, requiring constant iteration and testing. Fortunately, Microsoft has announced the upcoming release of?Mirroring, a modern solution for accessing and ingesting data continuously and seamlessly from any database or data warehouse into the Data Warehousing experience in Fabric. ?

Mirroring utilizes the database’s Change Data Capture technology, transforming it into appropriate Delta tables and landing it in OneLake. This allows for cross-joining of data across any database, enabling querying across any database, warehouse, or lakehouse. Every Mirrored database comes with default data warehousing experiences via a SQL Analytics Endpoint, which houses the metadata of the Delta tables and points to the data in OneLake. Power BI reports and semantic models can be built in Direct Lake mode, providing the fast performance of import mode without duplicating the data. ?

Mirroring will initially be available for Azure Cosmos DB, Azure SQL DB, and Snowflake customers, with support for other databases and data warehouses coming in 2024. This will enable organizations to access data faster and start building analytics and insights at a rapid pace. Additionally, the process can be completed by someone with experience with Power BI, rather than requiring a seasoned Data Engineer. The Mirror aspect will be near time, like technologies like Azure Synapse Link. Data is replicated within the OneLake system, conforming to your data retention policy. ?

From a business standpoint, Mirroring will improve speed and cost implications. Instead of building custom ingestion pipelines, you can turn on Mirroring and have access to data in mere minutes rather than days/weeks. Data engineers can start enriching and modeling the data sooner, delivering value to the business quicker. Reach out to discuss your specific needs and experience the future of data management with our?Fabric Modernization?proof-of-concept.?