How to Load a Fabric Warehouse?

Technology

The data warehouse in Microsoft Fabric was re-written to use One Lake storage.? This means each and every table in the warehouse is based on the Delta file format.? Staying away from single transactions will keep the warehouse performing at its best.? We already learned in a prior article that T-SQL insert statements can be used to add rows to a table.? Are there any other ways to insert or update rows in our warehouse?

Business Problem

Azure Data Factory has been a popular Extract, Load and Translate (ELT) tool.? Data pipelines exist in the Azure Data Factory, Azure Synapse, and Microsoft Fabric services.? Therefore, we are going to explore how to mesh data from Amazon Web Services (AWS) S3 buckets, Azure Storage Containers, and Google Cloud Platform (GCP) buckets into a final table in our warehouse.?

The Standard & Poor's 500 is a stock market index tracking the stock performance of 500 or so of the largest companies listed on stock exchanges in the United States.? We will be using five years’ worth of data which equates to over 2,500 comma-separated values (CSV) files.? The performance of the Data Pipeline will be compared to the new COPY INTO statement.? Let the best algorithm win on speed of execution.

Results

The output from this article is a comparison between all three cloud vendors when copy data from storage into the One Lake. Please see the article on SQL Server Central for full details.