Data Platform News (August 2024)

Welcome to the first edition of this newsletter with a little "rebranding"!

At the end of July I left Microsoft and joined InfiniteDATA Services as a CTO. This doesn't mean I've completely abandoned the Microsoft technology stack. I am broadening my horizons and now, in addition to Microsoft Fabric, my scope of interest includes other data platforms with a significant presence on the market. With that, I decided to: 1) resume the newsletter (I paused it at the end of 2023), 2) expand the scope of the newsletter to more data platforms (I'll start with three) instead of just one, 3) not separate community content from the content produced by vendors, 4) try to keep up to 10 links per platform per month (super hard to achieve!).

So, starting this month you will see me posting subjectively selected news from Snowflake, Microsoft Fabric and Databricks (disclaimer: I'm biased towards some of these technologies ;-)). As always, every month the list of links in my newsletter is going to be absolutely subjective and based on what I read, watch, listen to or encounter with customers.

So, without further ado, let's get to what happened last month.

Snowflake

• Snowflake announced the preview of Cortex Analyst, a fully-managed Snowflake Cortex feature that enables you to create applications in which users can ask questions in natural language and receive direct answers without writing SQL. Although the feature is fully managed, it requires some setup - you need to create a Streamlit app and a semantic model (.yaml file) stored in Snowflake stage. Snowflake provided a nice Quickstart (and a video with this quickstart's walkthrough) that shows how to get started with Cortex AI. Also, you can read more about what's behind the scenes: Snowflake Cortex Analyst: Behind the Scenes.

• Speaking about Snowflake Cortex, we got general availability of cross-region inference for Snowflake AI & ML features. Now you can process inference requests in a different region if the request cannot be processed in the region where the inference is originally requested (that can help if your Snowflake account is in a region that does not support some of the features).
• Snowflake continues their investments in out-of-the-box performance optimization and cost reduction (e.g ingestion of JSON and Parquet files quicker by 25%, up to 40% improvement in query efficiency, reduced cost for Automatic Clustering, Search Optimization and Materialized Views). Read more here: Snowflake Innovates on Performance & Efficiency While Reducing Costs. Also, some folks reported that the cost of serverless task has been reduced as well (check in Credit Consumption Table).
• Snowflake also continues to invest in the SQL surface area. Sample release notes from August: RANGE BETWEEN window frames with explicit offsets (GA) and Wildcard filtering for functions, new functions supported by RANGE BETWEEN with explicit offsets and more.
• As a part of expanding their data governance and management offer, Snowflake announced a preview of Differential Privacy. Differential privacy is a standard for data privacy that limits the risk that a user could leak sensitive information from a sensitive dataset. It provides strong protection against re-identification that is particularly effective against targeted privacy attacks. Learn more on this new feature by watching the video on the Snowflake Developers YT channel: Differential Privacy In Snowflake: How To Share More With Stronger Protections.
• Let's not forget old good data warehousing is still there in Snowflake. Read a blog post on three new ways to analyze pruning in Snowflake by Tim Sander . The blog post presents ways (some useful code provided!) to analyze pruning using TABLE_PRUNING_HISTORY, SEARCH_OPTIMIZATION_BENEFITS, and TABLE_DML_HISTORY account usage views.
• From Snowflake community, I got back to the lecture of blogs by John Ryan . Here are some of the latest blog posts from John full of practical information: Snowflake Dynamic Tables: Key Insights, Boost Snowflake Speed with Query Profile Mastery, Snowflake: Time Travel, Fail-safe, Data Keep, Choosing the Right Snowflake Warehouse: It's not Rocket Science, and Snowflake Virtual Warehouses: Performance vs. Cost. I especially enjoy John's way of presenting features through customer's benefits perspective (how customers can save money).

Microsoft

• August started with an important announcement from Microsoft. They announced the availability of trusted workspace access and managed private endpoints in any Fabric capacity. Until August these security features required F64 or higher capacity. In addition, managed private endpoints can be used with Fabric Trial capacity as well. For a table listing Fabric features according to SKU type check this documentation page: Microsoft Fabric features by SKU.
• Another big news for Microsoft Fabric was introducing AI Skills in Public Preview. Similarly to Cortex Analyst in Snowflake, AI Skills in Fabric allow you to create your own conversational Q&A systems using generative AI producing SQL queries. Microsoft came up with a nice tutorial on how to implement AI Skills over a well-known Adventure Works DW dataset - see here: AI skill scenario (preview) (the code from this article can be useful if you want to download Adventure Works DW dataset to any other platform - I used it with Databricks and Snowflake). Also, some community folks immediately posted useful blog posts showcasing ideas on how AI Skills can be used in code-based approach (notebooks) or in combination with prebuilt AI services in Fabric. Read these three blog posts by Sandeep Pawar : Use AI Skills As Cell Magic In Fabric Notebook, Asking Multilingual Questions To AI Skills Using Azure AI Services and AI Skillet : Local Fabric AI Skills App (the last one describes how to access AI Skills from your local web browser!).

• Of course, AI Skills was just one of many new features that landed in Fabric in August. In the Microsoft Fabric August 2024 Update you will find a full list of them, including: data limit in Power BI (maximum data load for a single session per visual), dynamic per recipient report subscriptions, delivery of report subscriptions to OneDrive and SharePoint, DAX query view in the web, TRUNCATE TABLE for warehouses, updates to manageability of the underlying Delta Lake for warehouses (enabling/disabling V-Order, pause/resume Delta Log publishing), new and updated connectors in pipelines (Salesforce, Vertica) and Dataflows Gen2. Also, you can watch the Fabric Monthly Update | August 2024 video on the Fabric YouTube channel - it's packed with demos of the new features (I love this format even though the whole video is almost 1h long!). One important note: make sure you update your Fabric notebooks and use NotebookUtils in place of MsSparkUtils (new features will only be added to the NotebookUtils namespace).
• One thing was a part of August update (in the Data Science section?!), but deserves to be mentioned separately. Semantic Link Labs, developed by Michael Kovalsky, is now a part of official Microsoft's GitHub repo and open sourced. The new version comes with tons of new functions, including a lot of useful functions for Power BI developers (you can run Best Practices Analyzer on semantic model, translate model metadata on the fly between different languages, get report definition and more). Check it out here: Releases · microsoft/semantic-link-labs. If you are not familiar with Semantic Link Labs, it is a python library intended to be used in Microsoft Fabric notebooks and containing many useful functions in the realm of semantic models, reports, lakehouses and anything Fabric-related.
• As much as I wouldn't call myself a huge fan of Dataflows Gen2 in Fabric, I value the new article on Fabric documentation describing implementation of SCD type 2 with this Fabric item: Slowly changing dimension type 2. The article contains a link to the Power Query Template that has the full Slowly changing dimension type 2 pattern solution. Great to see more and more articles on patterns and practices in Fabric documentation.
• Speaking about patterns and practices, Sean M. wrote a very useful blog post Exploring CI/CD Capabilities in Microsoft Fabric: A Focus on Data Pipelines describing how to combine different features and products (Azure DevOps, Fabric git integration, Deployment Pipelines in Fabric) into a basic CI/CD pattern for Fabric.
• On the same CI/CD topic, Kevin Chant presented a blog post on naming conventions for git repositories and branches: Naming conventions for Microsoft Fabric Git integration repositories and branches. Make sure you check our other blog posts from Kevin as CI/CD and DevOps (or rather "KevOps") are his strong specialties.
• Another useful blog post from Fabric community member, Gilbert Quevauvilliers : Downloading Scanner API data using a Microsoft Fabric Notebook. Gilbert described how to use Fabric notebook to ingest Scanner API data into a lakehouse (for example to automate tenant monitoring). Cool stuff!
• Chris Webb published several interesting blog posts last month. My two favorite ones are: Which columns are loaded into memory when in Power BI Direct Lake mode? and Monitoring the opening of DirectQuery connections in Power BI. Both super practical if you use (or plan to use) DirecQuery and/or Direct Lake mode.
• An interesting blog post on Implementing Data Vault 2.0 on Fabric Data Warehouse authored by Michael Olschimke and Trung Ta was published on the Analytics on Azure Blog. You can find code examples on how to implement standard data structures of Raw Data Vault in Fabric warehouses.

Databricks

• AI/BI Genie in Databricks is a competitive feature for Fabric's AI Skills and Snowflake Cortex Analyst. Recently, Databricks posted a blog post Onboarding your new AI/BI Genie describing a day in the life of a Genie so that users can get to know how to use it and what to expect. Also, you may want to check out a 40-minute video Introduction to AI/BI Genie - A New Conversational Analytics Experience for Business Users presenting Genie fundamentals, key concepts and extensive demo.

• New runtimes (15.4 LTS and 15.4 for Machine Learning) were announced generally available in August - here are the release notes for Azure Databricks: Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS for Machine Learning.
• A long awaited For each task type in Workflows is generally available - see release notes. Now it's easy to run parametrized nested task (e.g. a notebook) in a loop (with configurable parallelism of loop iterations). Read more: Run a parameterized Azure Databricks job task in a loop.
• Lakehouse Federation became generally available - see release notes. Lakehouse Federation in Databricks allows you configure read-only connections to popular database solutions (including Snowflake, Azure Synapse and SQL Server) and run federated queries combining data stored in multiple different platforms (with support of pushdowns and Single Sign-On in specific cases). Databricks uses Unity Catalog to manage query federation.
• Unity Catalog is a beating heart of Databricks. No wonder the company invests a lot in making customers aware of recommendations and best practices to get the most out of the key feature. Read the Azure Databricks documentation article: Unity Catalog best practices (you can find this article also for Databricks on AWS and GCP).
• Responsible AI becomes more and more hot topic looking at high interest of organizations to benefit from RAGs, LLMs and all the AI-related stuff. Databricks presented an interesting blog post discussing key challenges in responsible AI and how to approach them: Responsible AI with the Databricks Data Intelligence Platform. Even though I realize the blog post positions well some of the Databricks features, I consider much of the advice from the article as universal and applicable to any data platform.

• Dustin Vannoy published a video Databricks CI/CD: Azure DevOps Pipeline + DABs that shows most of the steps involved in setting up automatic deployment using Azure DevOps and Databricks Asset Bundles. You may also check out older videos from Dustin on introduction to CI/CD for Databricks and advanced examples of CI/CD, plus read the blog post Integrating Databricks Asset Bundles into a CI/CD Pipeline on Azure by Alfeu Duran where you can find useful diagram and code examples.

Sources of news and updates

Here are some links you may find useful to stay updated on what's going on in all three platforms:

Snowflake

• Inside the AI Data Cloud | Snowflake Blog
• What’s New | Snowflake Documentation
• Snowflake Inc. - YouTube
• Snowflake Developers - YouTube
• Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science (medium.com) (reading selected posts may require paid membership)

Microsoft Fabric

• Microsoft Fabric Blog
• Power BI Blog—Updates and News | Microsoft Power BI
• Microsoft Fabric release plan documentation - Microsoft Fabric | Microsoft Learn
• Microsoft Fabric - YouTube
• Azure Synapse Analytics - YouTube

Databricks

• Databricks Blog
• Azure Databricks release notes - Azure Databricks | Microsoft Learn
• Databricks platform release notes | Databricks on AWS
• Databricks release notes | Databricks on Google Cloud
• Databricks - YouTube
• Databricks Platform SME – Medium (reading selected posts may require paid membership)
• DBSQL SME Engineering – Medium (reading selected posts may require paid membership)

My quick summary

In my opinion, August was a month of conversational self-service analytics features. Cortex Analyst, AI Skills, Genie - they all serve one purpose: take user's prompt, convert it into executable SQL code and return meaningful results to the user. Some fundamental questions came to my mind after I played a bit with all three features and had my first observations: 1) Who are these things for? 2) How can the quality of SQL code generated by LLMs be guaranteed? But these are topics for another article or a few of them ;-)

Besides "the AI-based-text-to-SQL battle", I would highlight: 1) Snowflake continuously investing in out-of-the box performance, the SQL surface area, security and data governance, 2) Microsoft listening to customer feedback and making security features available for all SKUs, 3) Databricks continuing to set product gravity to Unity Catalog (Genie requires data used by the feature to be cataloged in UC) which seems to be reasonable as having data centrally governed and managed can be beneficial for customers working on applications for AI.

As always, if you found some interesting news or articles last month, please share them with me and the readers of this newsletter. Thanks for reading and until next time.