The Future of Metadata, Data Council in Austin, and more!

Welcome to this week's edition of the ? Metadata Weekly ? newsletter.

After a short break, I am back to bringing you my recommended reads and sharing my (meta?) thoughts on everything metadata!

But first, I have something to share: Atlan announced a $50M Series B led by Salesforce Ventures, Sequoia, and Insight Partners. And we’re moving towards our next phase of growth, continuing on our mission to help data teams do more. I am very excited about what we have built and looking forward to Atlan’s journey ahead. ?

Also, I’m going to be speaking at Data Council in Austin next week, our first in-person data community event in a long time. I’m excited to extend an exclusive invitation to readers of Metadata Weekly to join me (and our team at Atlan along with our partners at Great Expectations) to watch 1.5 million bats take off at sunset, in a special human of a data sunset cruise. ??

? Spotlight: The Future of Metadata

For the past year at Atlan, we’ve been working closely with our customers and thinking long and hard about the future of metadata in the modern data stack.

While the data stack has solved some important problems, it’s become increasingly chaotic and siloed. The experience of working with these tools today is fundamentally broken. Benn Stancil?wrote about this?and made an open call for the “Modern Data Experience”. While I don’t fully know what the future of the modern data stack holds, I do know that metadata will be key to stitching together the ideal data experience.

Metadata was all the rage last year. Yet all the hype and debates focused on metadata architectures, push vs pull, data discovery vs data catalogs, etc. We believe these are the wrong problems to be focusing on.

“We believe that the next leap forward in the metadata space will not come from perfecting the architecture. Instead, it will come from helping data teams get real value from the gold mine of metadata that’s generated today.”

We have still barely scratched the surface of what metadata can do. Metadata holds the key to solving dozens of problems in our space —?allocating compute resources dynamically based on usage metadata, scoring data health and insights, purging stale data assets, and even dynamic pipeline optimization.

For example, imagine a world where these situations are the norm:

• When a data quality issue is detected in a source table, the system?automatically stops the downstream pipelines?to ensure that incorrect data doesn’t make its way to the dashboard.
• The system leverages past usage logs to?automatically tune data pipelines?and optimize compute performance by shifting loads towards frequently used data assets and optimizing the schedules of data pipeline runs.

We believe that the solution to this is moving from today’s passive, siloed systems to truly?activating metadata. Think of a?Segment?or?Zapier-like experience where metadata can create harmony across tools and power perfect experiences across the data stack.

To make this future happen, we need to open up metadata and move from augmented data catalogs to “metadata anywhere” orchestration systems.

?? Recent Fave Links

Solidarity in the small stuff by Emily Thompson

“When rolling out a new idea to an organization, I always test it first with a small subset of the team. It’s usually difficult to get the tone right on the first try. Just like shipping an MVP, you can learn where the rough edges are in practice with a small test group and find areas to improve before committing to the idea at a larger scale, which is more costly if it doesn’t work out. Drumming up excitement with a smaller team is also easier, and that excitement can be contagious when rolling it out to the broader organization. I did that with Doc Days, as well as half a dozen?data team branding initiatives.”

While Emily writes about some amazing culture stack initiatives (like Doc Days, as covered in this post), I want to highlight something that she talks about in passing, which I think is actually more important than the ritual itself: the approach to figuring out the right cultural rituals for your team.

Here’s the thing about culture: every team is different, and you can’t just take a cultural ritual that worked for someone else and expect that it will work for your team. This is where I’ve found the “MVP” approach really shines: Align your team to the “problem” that you’re looking to solve, and “experiment” with the approach or ritual. Start with a small group of people and an informal start. Observe how people react. Did it naturally catch on? Did the team enjoy it? What was their feedback? Did it help achieve your overall objective? Was it fun?

If your experiment didn’t work, tank it and go back to the drawing board till you find a ritual that works! If it was, find a champion from the early group of “testers” who’s naturally excited to scale it and make it a ritual. Remember, going from early success to a ritual — something that is part of your DNA as a team takes consistency and time.

Special Mention ??

Sam’s Data Detective Bingo card pulled so many triggers for me! ??

On that note, what are the other fun data games you’d recommend? I’m following Jason Ganz’s tweet with a lot of excitement!

???More from My Reading List

• Data Collection is Hard. You Should Try It. by Geroge Ho
• Building a Data Engineering Center of Excellence by Richie Bachala
• Recipe for building your first Data Product in a Data Mesh by Thinh Ha
• KPIs Every Data Team Should Have by Jesse Anderson
• Data Team Branding by Emily Thompson

And some more that I added to my data stack reading list. If you haven’t yet checked out the list, you can find and bookmark it?here.

???Bat & BBQ on a Boat: For the Humans of Data in Austin

Turns out that free-tailed bats like to hang out in Austin! And at sunset, 1.5 million bats emerge from the bridge at the same time. We were told it's a spectacular sight, and this time of the year also happens to be the best time to experience this.

So we thought, what better way to celebrate the first in-person data community event of 2022 other than by hosting a special boat cruise to experience all the goodness that Austin has to offer?

Atlan and Great Expectation are coming together to host a special Bat Cruise for the humans of data on the 24th of March.

In case you are around in Austin for Data Council on 23-24 March, fill out this form for an invite to join us. ??

I'll see you next week with more interesting updates from the modern data stack! ???Meanwhile, you can subscribe to the?newsletter on Substack and connect with me on LinkedIn?here.