ETL + Metric Store = Awesome

This could be significant. This morning, dbt announced that it will acquire Transform, the people behind MetricFlow and the metric store concept. Just last week I wrote about how we needed ETL tools and the semantic layer, or metadata to come together.

For some background, there's a nice write-up by the good folks at Hex that covers the rise of the modern data stack, dbt's journey toward a metric layer, and why joining forces with Transform makes so much sense. To summarize:

• With the advent of the Modern Data Stack around 2010, as teams started using Redshift, Stitch, Fivetran, etc. to manage data, Looker introduced LookML, a DSL for defining business-ready data. In many ways, it was the first metrics layer.
• Around 2016 dbt arrived, allowing data teams to build complex transformations in SQL right in the database. A standalone semantic layer was created, but without support in the front-end analytics products, the idea of a universal metrics layer languished.
• It makes a ton of sense for metrics to live at the transformation layer, and by joining forces with Transform, dbt can accelerate that journey.

Semantic layers have traditionally been integrated into business intelligence tools, but defining metrics in conjunction with transformations, and then opening up that metric layer to all analytics tools, makes a lot of sense.

Last week I wrote: "We are missing a widely adopted standard for metadata that would allow data catalogs and data pipeline tools to interoperate."

So you can see why I think this announcement by dbt is significant. Today's dbt announcement blog ends with their commitment to an open standard for the semantic layer.

This could be the catalyst that the industry needs to adopt a metadata standard. I predict fast progress, and a few more announcements, in this area over the next few months.