Special Metadata Weekly Edition from Atlan ?
Prukalpa ?
Co-Founder at Atlan –?Home for Data Teams | Forbes30 & Fortune40 lists | TED Speaker
Welcome to this week's edition of the ? Metadata Weekly ? newsletter.
Every week I bring you my recommended reads and share my (meta?) thoughts on everything metadata! ??If you’re new here, subscribe to the?newsletter?and get the latest from the world of metadata and the modern data stack.
But this edition is different, and I couldn’t be more excited to share this update with all of you. ??
Over a year ago, we started re-platforming Atlan to help us take the next leap forward in metadata — from a traditional, passive, catalog-based approach to truly activating metadata and bringing it back into teams’ daily workflows, helping drive value with metadata. Last week, we announced Atlan’s BIGGEST update to help data teams leverage the power of active metadata to enable a DataOps team.
Tomorrow, my co-founder, Varun and I unveil the all-new Atlan during our FIRST EVER public town hall. I’d personally love to see you there, bring in your questions, share your feedback, or drop by to just say hello. You’ll receive an invite?by signing up on this link.
??Spotlight: What’s New in the ALL NEW Atlan?
Last year, Gartner released their?Market Guide for Active Metadata?and declared that “Traditional metadata practices are insufficient…” This year, Forrester’s traditional "Machine Learning Data Catalogs" report evolved into the “Enterprise Data Catalogs for DataOps” report, marking the arrival of a category and?recognizing the need for automation, collaborative flows, and personalization in modern metadata platforms?(ICYMI:?Atlan was named a Leader in this report, scoring the highest possible score in 17 of the 26 evaluation criteria).
In the last year, our team has worked closely with our customers and some amazing data leaders to completely reimagine Atlan and help data teams unlock the true potential of metadata with a few key foundational capabilities. For readers of this newsletter, once you see the updates, you’ll know WHY I’m so excited!
Personalization (Netflix for Data, anyone?)
Data teams are diverse, and context means different things to different types of users — Let’s understand this with something as simple as a “table”.
???For a data engineer, the context that matters is, where does this data asset come from? Are the connected pipelines working or did they break? Was the data delivered as per SLAs?
???For a data analyst, the context means something fundamentally different. The analyst wants to know, what the column names mean. Are there missing values? What is the frequency distribution for this variable?
Similarly, this context is different for an analytics engineer, marketing analyst, and business user. So why do we still serve the same generic experience for every data user?
In the last year, we’ve been thinking deeply about this problem — and taken inspiration from consumer software! When Netflix can serve you and me incredibly personalized experiences, why can’t we do the same for our data users?
Introducing?Personas and Purposes?— the first step in this journey we are taking to build a personalized home for every data user.
Activating Metadata into Embedded Collaboration Workflows ??
In one of the previous editions of the newsletter,?I talk about the concept of flow. But here’s the thing — data teams are SO diverse, and our tooling stack is SO diverse that flow becomes almost impossible to achieve. Check a message on slack, get a ticket on JIRA, open up a pipeline on Airflow, go to dbt to make a change to your model - I could go on about the painful day in the life of a data practitioner. Embedded collaboration is about work happening wherever you are, with the least amount of friction, saving data practitioners from the endless tool- and context-switching. We believe that "Active Metadata" holds the key to making these experiences possible – for metadata to flow effortlessly and quickly across the entire data stack, embedding enriched context and information in every tool in the data stack.
Here are a few examples of what embedded collaboration could look like:
???When you ask a question about a data asset in Slack (”how do we define customers again?”), our bot brings context about that asset directly to you in Slack.
???When you’re in a BI tool, and say “can I trust this dashboard”, our chrome extension can give you context about the dashboard right then and there — “yes, the connected pipelines ran successfully today & it marked as a ‘verified’ dashboard by the data analyst team
领英推荐
???When you’re browsing through the lineage of a data asset and find an issue, you can create a Jira ticket right then and there
Check out our product page for a dozen more micro-flows that are now live on Atlan?and?watch the launch video here.
Activating Lineage ??
Our product team has worked pretty hard to bring the concept of “embedded collaboration” to every aspect of metadata. Here’s the challenge with lineage graphs: they are pretty to look at, but difficult to act on. But actionable lineage can actually be really powerful — it can help drive powerful use cases from root cause analysis and issue tracking, to impact analysis and notifying downstream users of changes.
This is why we’ve invested significantly into building activation capabilities into Atlan’s column-level lineage capabilities:
?? Spot an upstream issue via lineage graph, create a JIRA ticket
?? Did the pipeline break upstream? Attach an announcement to impacted downstream assets, and send a message on slack
?? Mark a column as “confidential” and propagate the tag to all downstream derived columns
DataOps Workflows ??
Last year, I wrote about the concept of a metadata lake, and shared, that activating metadata holds the key to dozens of use cases like alerts and notifications, cost management, remediation, security, programmatic governance, and auto-tuned pipelines, and more. With this update, we have taken a BIG leap towards making many of these use cases come alive and look forward to co-creating more with our amazing customers. ??
And dozens of other updates! ??????
Tomorrow — Varun and I will not just show you the new Atlan, but we’ll walk through the key learnings we’ve had over the years and the fundamental insights that have driven the all-new Atlan.
???Rise of Active Metadata
If you’re looking to explore how active metadata is becoming the driving force behind a lot of innovation in the modern data world – augmented data catalogs, autonomous DataOps, data fabric, and data mesh, data governance, and consumerization of data tools, check out these resources:
?? Special Invite for Metadata Weekly Readers
Over the last few months, I have loved sharing my learnings and insights about building and working with amazing data leaders with all you amazing readers of the Metadata Weekly newsletter. I personally would love to have you join us tomorrow as we share what our team has built over the last 18 months and talk about the future of active metadata and DataOps as we see it. ??
If you cannot join but would like to learn more,?ping me on LinkedIn?and I’ll be happy to share the recording of our first public townhall with you.
Senior Data Engineer | Azure | Databricks | Spark | 7 Certifications
2 年I'm doing market research to suggest a good data governance tool for the companies I work for, Atlan has caught my attention a lot.
APAC Commercial Leader, Enterprise Solution Sales, Strategy & Operations, Go-To-Market, P/L | Healthcare, Technology (SaaS)
2 年Fantastic!
SVP Client Insights Analytics (Digital Data and Marketing) at Bank Of America, Data Driven Strategist, Innovation Advisory Council. Member at Vation Ventures. Opinions/Comments/Views stated in LinkedIn are solely mine.
2 年Very forward looking,would help users be aware of the different data assets and its dependencies. What if the client mostly has on perm data sources would this work or is the expectation that your data sources need to be in the cloud.