Workshop - 08/03/24
Building Modern Data Platform for Analytics
I want to thank my friend Ed Pollack for asking me to speak at SQL Saturday Albany. This full day pre-conference workshop is a bargain for those professionals who want to learn about the MDP. You get breakfast, lunch, and my half of a decade experience on creating Modern Data Platforms in Azure for various clients who want to do analytics.
Here are the details that will be covered that day. Please sign-up for the workshop using Eventbrite. Hope to see you this summer!
Many companies are placing their corporate information into data lakes in the cloud. Since storage costs are cheap, the amount of data stored in the lake can easily exceed the amount of data seen in a typical relational database. Regardless of the types of files in the data lake, there is always a need to transform the raw data files into refined data files for analytics, machine learning, and/or AI.
领英推荐
The Delta Lakehouse design uses a medallion (bronze, silver, and gold) architecture for data quality. We can we abstract the read and write actions in Spark to create dynamic notebooks to process data files. Data pipelines can be used to bring remote data into the lake as well as orchestrate data processing. A metadata driven design allows for the inputs to the dynamic notebooks to be stored in a central place.
The most important part of a modern data platform is security. Microsoft Entra, formally known as Azure Active Directory, can be used to secure the files in storage. This security layer is used in both the Apache Spark and Serverless SQL pools. Designers use a variety of tools for reporting. The Serverless SQL Pool turns a data lake files into a read only database tables. While the demos in this course are Azure specific, the concepts can be used with any cloud service.
Lessons:
Data Architect at Insight
8 个月There is less than 24 days before the pre-conference sessions for SQL Saturday Albany. Don't miss out training from IT professionals who have been using the technology for years. The cost includes training and food. Hope to see you in my class!