Cloud Data Platform Weekly

Hi there, well it's been a busy week for 2 of the cloud data warehouse/lakehouse vendors with both Databricks and Snowflake conferences. There's been some juicy announcements from Databricks which I'll highlight below. It's also probably been a busy week for Microsoft what with Fabric now 5 weeks out into public preview and a lot of expectation on it.

Let's get to it.

Using Domains in Fabric to group workspaces and isolate workloads

I've been looking at Domains within Fabric recently and started sketching out what it would look like with multiple business areas, and workspaces in those areas. Then looking at how to isolate workloads using different Fabric capacities attached to those workspaces for dev/test/prod.

Temp Tables in Fabric Lakehouse SQL Endpoint and Warehouse

Please note temp tables within Fabric are not officially supported yet, but I wanted to see what was/wasn't working in terms of functionality. I'll be updating this blog as and when things change. This blog also kicks off the Fabric SQL Series and more is coming soon. Read more here.

Things I've Been Reading and Watching This Week

First up it's Databricks, who dropped a whole lot of announcements at the Data and AI Summit which included my personal favourites:

• Delta 3.0 release including UniForm: Big news here in terms of the storage format wars with Apache Iceberg and Hudi...because UniForm in Delta 3.0 allows data written in Delta to be accessible as if it were Iceberg or Hudi. Huge. Read more here.
• Delta Liquid Clustering: Not sure how to layout/partition your Delta tables for optimal performance? Well, Delta Liquid will be your friend as it can automate the layout of data and ensures data isn't over or under partitioned. Read more here and also here.

In terms of Snowflake, the Container service looks interesting... Have a watch here.

Other Things

• Here's the GitHub link to Delta 3.0. https://github.com/delta-io/delta/releases/tag/v3.0.0rc1
• Sandeep Pawar has a great blog Fabric : Not All Delta Tables Are Created Equally in which Sandeep dives into different methods of creating Delta tables in Fabric and compares performance and quality. Read more here.
• I keep going back to Medallion Architecture documentation as it's the current hot thing right now with Fabric and how to lay out your data and ETL/ELT pipelines. Read Microsoft's definition here.

Have a great weekend, and I'll see you next week.

Andy