Day 1 of Coalesce 2024: dbt Labs Unveils a Unified Vision for Seamless Data Collaboration Across Platforms with AI-Driven Innovation

dbt Coalesce 2024 opened in Las Vegas to a crowd of 1,800+ data practitioners, leaders, and executives at Resorts World. Plus, thousands more joining in online. In the keynote, Tristan Handy , Founder & CEO @ dbt Labs, set the stage with a bold vision for the analytics development lifecycle. He shared dbt’s mission to make data and AI more accessible and trustworthy in fueling innovation.

He unveiled a new ethos they’ve termed “ONE dbt”. A concept that goes beyond specific features to encapsulate a holistic approach to data analytics. One dbt vision aims to unite platforms, users, and workflows under one framework. It will enable seamless collaboration across any data platform, in any cloud environment, while being powered by AI to accelerate insights and productivity.

One dbt: A New Era of Unified Analytics

At the heart of this vision is the belief that you shouldn’t have to make a choice between cloud platforms or between dbt Core and dbt Cloud. Instead, dbt Labs is building toward a future where teams of engineers, analysts, and decision-makers can collaborate under a unified system, using tools that best suit their needs. Once dbt embodies this integrated, scalable approach, it will support the full analytics development lifecycle (ADLC) across platforms. All from orchestration to cataloging and beyond.

Why One dbt

Several key trends have been driving the focus on integration and unification:

1. Platform Flexibility: Companies are adopting multi-cloud strategies, and dbt Labs recognized the need to provide a flexible platfrom. It focuses in providing seamless interoperability between platforms like Snowflake, Databricks, Redshift, and Athena.
2. Collaborative Analytics: Self-service BI tools and generative AI brings business users and domain experts closer to analytics workflows. This shift has opened new possibilities for collaborative data work.
3. The Trust Imperative: With issues like poor data quality and a lack of data ownership plaguing many organizations, there’s an urgent need to build greater trust in data across business and technical teams. The Analytics Development Lifecycle (ADLC) offers a framework to mature workflows and improve data quality, trust, and governance.

Major Announcements from Coalesce 2024

dbt Labs showcased several exciting new features and updates designed to support their One dbt vision:

1. Supercharge Collaboration with Cross-Platform dbt Mesh

One of the most anticipated announcements was Cross-Platform dbt Mesh, a feature that allows organizations with multiple data platforms to collaborate under a centralized, governed data model. This capability enables practitioners to reference and reuse data assets from different platforms. It helps in breaking down silos and ensuring that teams can work together regardless of their specific tech stack. Dbt currently supports Snowflake, Databricks, Redshift, and Athena. This feature brings the promise of data mesh into reality for large enterprises that will truly help you to maintain a unified workflow across diverse data estates.

2. Enhanced Query Performance via Iceberg Table Support

As a critical enabler for Cross-Platform dbt Mesh, dbt Cloud now supports Apache Iceberg (a popular table format for handling large-scale data lakes). By supporting Iceberg, dbt allows you to benefit from enhanced query performance and schema evolution while maintaining the familiar dbt workflow. dbt is also currently working on Iceberg support for Snowflake.

NOTE: dbt currently supports Athena, Spark, Databricks, Starburst/Trino, and Dremio

3. True Data Democratizing through Visual Editing Experience

dbt Labs is democratizing data access by empowering a broader range of users with a new Visual Editing Experience. This drag-and-drop interface (currently in beta) is designed to help non-technical users contribute to the analytics workflow without needing SQL skills. This new interface allows you to safely build data products while adhering to governance standards, relieving pressure on overburdened data teams. For SQL experts, the tool also offers a visual representation of complex models, helping to optimize and explore data pipelines more intuitively.

4. Boost Productivity withdbt Copilot

Another exciting feature is dbt Copilot, the embedded AI engine that supports various workflows across dbt Cloud. dbt Copilot helps automate repetitive tasks like test generation, documentation, and semantic model creation. The Copilot also includes an AI-chatbot powered by the dbt Semantic Layer. It is also integrated with Snowflake and also supports integration with OpenAI via custom API keys. Companies leveraging Copilot will see significant boost in productivity and data quality.

NOTE: dbt Copilot is currently in beta

5. Advanced Continuous Integration (CI) with Compare Changes

To enhance data quality and prevent unexpected errors in production, dbt introduced Advanced CI with Compare Changes, now generally available for dbt Cloud Enterprise customers. This feature enables teams to see the exact changes made to data models, including columns and rows, directly within their Git provider’s pull request interface. This transparency helps teams identify potential issues before deployment, reinforcing trust in the analytics process.

6. Improve Efficiency with Auto-Exposures for Tableau

In another significant update, dbt Cloud now supports Auto-Exposures for Tableau, with Power BI integration coming soon. This feature automatically populates downstream dashboards with exposure data. This push ensures that stakeholders are always working with the freshest, most accurate information. You get full visibility into how data models are being used across dashboards. If you are technical user then you can prioritize your work based on impact. If you are business users, then can trust that you are making decisions based on most up-to-date data.

7. Increase Trust via Data Health Tiles

To further enhance trust in analytics, dbt Cloud now offers Data Health Tiles, which can be embedded in dashboards to provide at-a-glance insights into the quality and freshness of the supporting data. These health signals ensure that decision-makers have quick access to the status of their data assets and can navigate back to dbt Explorer for further investigation if needed.

New Integrations and Open Source Advancements

dbt Labs is growing its system with new integrations. AWS Athena (now generally available) and Teradata (in preview) are supported, and Athena will be one of the first platforms to work with Cross-Platform dbt Mesh. dbt also added a new connection to Power BI through the dbt Semantic Layer.

On the open-source side, dbt Core v1.9 includes updates like the new microbatch strategy to help process large datasets more efficiently and easier setup for snapshot configurations. These changes show dbt’s continued dedication to providing useful tools that help solve big data challenges.

Conclusion

Day 1 of Coalesce 2024 showcased dbt Labs’ commitment to empowering data teams, enhancing collaboration, and building trust in data with its bold “One dbt” vision. This unified framework aims to bring together platforms, users, and workflows for seamless collaboration across any cloud environment, driven by AI. From the Cross-Platform dbt Mesh to the new Visual Editing Experience, the announcements at Coalesce 2024 highlight dbt Labs’ dedication to its customers and opensource users.

You can read the article by IntellaNOVA on Medium:

https://medium.com/@ibbyrahmani/day-1-of-coalesce-2024-dbt-labs-unveils-a-unified-vision-for-seamless-data-collaboration-across-b5a8add27a3a

Shoutout to the other presenters at the keynote:

dbt Labs #dbt #dbtlabs #datatransformation #dataquality #datamesh #datademocratization #datamanagement Sunil Mallam

Shoutout to the other presenters at the keynote:

Amy Chen ,?Product Manager?@?dbt Labs

Greg McKeon ,?Staff Product Manager?@?dbt Labs

@roxanne dahlke( pourzand Product Manager?@?dbt Labs

James Dorado ,?VP, Data Analytics?@?Bilt Rewards

Tobias Humpert ,?Siemens Data Cloud Product Owner?@?Siemens AG

Yannick Misteli ,?Head of Engineering?@?Roche