Data Platform Weekly

Hello! Well, after a personal hiatus to get some boring home stuff sorted, DPW is back and I've got a bumper issue here! Time is money and all that so let's get going :)

ServerlessSQL Blogs

Over on my ServerlessSQL.com blog I've been concentrating on Synapse Analytics permissions and source control. With Fabric I've been looking at cloning tables in Warehouses, and extending one of the excellent Microsoft End-to-End Lakehouse tutorials.

• Synapse Analytics Source Control with Serverless SQL Pools Databases - how to connect a Synapse workspace to Azure DevOps and use Azure Data Studio to source control Serverless SQL Pools using the same repo (read here)
• Azure Synapse Analytics Custom Roles - how I used Azure Custom Roles to augment a fine-grained Synapse permissions issue (read here)
• What Happens When You Clone A Fabric Warehouse Table? - cloning a Warehouse table is really useful for dev, test, and back-ups, but what happens in the storage layer? (read here)
• Fabric Lakehouse Loading using Data Pipelines & Notebooks – Inspired by MS End-to-End Tutorials - or more accurately, how I overcame my fear of PySpark by taking one of the end-to-end tutorials by Microsoft and modifying it for my own requirements (read here)

Data Toboggan Synapse Analytics and Fabric Conference

Free registration for our next virtual Azure Synapse Analytics and Fabric conference on Friday 24th November is now live! This conference is actually split over 3 time zones so you can register for the relevant time-zone, or all 3! We will be announcing the schedule soon...

Each session is 20 minutes long and will cover areas of Synapse, Fabric, Power BI, and other Data goodness.

Register now

Speaking at #Data Weekender 6.5 - Saturday 4th November

I'll be speaking at the next #Data Weekender 6.5 next Saturday 4th November (9:15 am) about Fabric Warehouses, and the journey of the analytical SQL engine. Come along if you want to know about the history of the engine, and where we are now with Fabric. There are multiple session tracks on the day, so plenty of great sessions.

Check out the schedule

Register for free now

Things I've Been Reading and Watching This Week

• The monthly MVP Fabric Corner for October is now available, containing a whole bunch of content across all things Fabric. Check it out here
• A really good live stream session with Marthe Moengen and Christopher Wagner, MBA, MVP on Chris's KratosBI YouTube channel about Citizen Data Engineering. One to watch if you are looking at how de-centralised data engineering can work in the Fabric eco-system. Watch now
• There is an excellent YouTube channel Learn Microsoft Fabric with Will by Will Needham . I've been watching the Learn Apache Spark playlist that Will did through September. Definitely check it out if you want to get stated with Spark in Fabric. Watch now
• Pragati Jain has been posting about Exploring the Fabric SQL Endpoint with the Model view - essential if you want to understand how the Power BI modelling process works with Lakehouses. Read now
• I have an idea over on the Fabric Ideas website about adding schema advancement and append as an option when writing to a Fabric Lakehouse table using Data Pipelines. Please add your vote to it, if this is something you feel would be beneficial. Vote now
• The Azure Diagrams online diagramming tool now has Fabric services in, it's a really great tool for designing a basic architecture. Start designing now

I hope you have a great weekend and I'll see you next week.

Andy