Journey to Data Mesh ??

It’s a problem any student can relate to. You’re learning a new concept, and you’re stuck on how best to answer a question, or even understand altogether. It’s the universality of this experience that’s made Brainly the world’s most popular education app, serving students, parents, and educators, helping them answer questions and strengthen skills across subjects like English, Math, Science, and History.

Serving hundreds of millions of users, across diverse personas and subject matter, Brainly’s business generates an astounding amount of data, and their organization depends on it to better serve their users and continue their growth.

In this week’s newsletter, we spotlight how Brainly transformed its data and analytics strategy.?You can read the full story here →

? Spotlight:?Brainly’s Journey to Data Mesh

Illustrating some of what’s worked well for Brainly’s data team, Kasia Bodzioch-Marczewska, their Domain Lead of Data Engineering, joined Atlan at the 2023 Gartner Data & Analytics Summit in London, sharing the progress her team has made on Data Mesh, how Active Metadata Management can help drive necessary cultural and technical shift, and key takeaways for data leaders considering a similar journey.

Decision on Data Mesh

Recognizing the potential of activating Brainly’s data, their team began to consider implementing Data Mesh to better organize work, and encourage full ownership and stewardship of their data.

A?data mesh?is a?technical?and?cultural?approach to building a decentralized architecture that organizes data by a specific business domain, providing more ownership to data producers.

The Brainly team began their Data Mesh journey by defining two critical dimensions to master. First was technical, investing in technology that would better enable transparency, team-to-team collaboration, and data quality standards. Second, and perhaps more importantly, was cultural, enabling ownership and accountability, despite a decentralized governance model.

Further enhancing the value they could yield from Data Mesh, and treating data as a product, Kasia and her team found important alignment with the way Brainly’s product teams were organized and run.

“A few years back we decided, as a company, to decentralize our product teams,” Kasia shared. “Every product team at Brainly is independent and has their own tools and data.”?

While this model paid dividends for innovation and agility, making Brainly the leader in education technology that it is today, the siloed nature and ownership of this data meant frustrating back-and-forth whenever one team needed another’s data.?

“If we think about this data and this setup, if a program department like Tutoring, for example, wants to utilize financial data, they would have to go and ask. There was a very long process to get access to the right data, and to figure out which data you could use for your analysis,” Kasia explained.

The Data Mesh concept, combined with Brainly’s unique model of product domain ownership, was a clear and exciting opportunity. But to eliminate the team-to-team collaboration friction inherent to this strategy, Kasia’s team evaluated the Active Metadata Management market in search of a solution.

Data Catalog – Enhance Data Mesh

“We figured out that we needed a data catalog to support us with the cultural piece of Data Mesh. We went through several proofs of concept, and through those, we chose Atlan as our data catalog.”

In the early stages of their Data Mesh journey, Atlan has proved to be a crucial partner as Brainly migrates to a new data platform that will better support their new way of working.

“We’ve onboarded into Atlan all of our data sets from both our legacy platform and our new data platform” Kasia explained. “And for migration purposes, we’re using Atlan to identify these sources of the products that are to be migrated to the new data platform, as well as to figure out the downstream objects affected by decommissioning in the legacy platform.”

Brainly’s new data platform consumes a vast array of data sources, both structured and unstructured, passing through to a raw data lake in AWS S3, Spark and Glue for processing, through to a data mart using Snowflake, and Tableau or Metabase for visualization and analysis.

As users migrate from legacy-to-modern, Atlan serves as their gateway to understanding and applying data and the new platform’s capabilities. “Obviously, we have users consuming all of this. We have all of our assets in Atlan; everything integrated into Atlan. Basically, Atlan is the place where all data sharing happens,” Kasia explained.

Ten domain teams at Brainly are already using their new data platform, and increasingly depend on Atlan for crucial context about available data. “We see more and more people defaulting to the data catalog instead of going through a lengthy process of going back-and-forth (with questions),” Kasia explained.

Read the lessons learned by Kasia the data team at Brainly in this journey.

?? From Our Reading List

• Data Scientist: the most ambiguous role of the 21st century?by Remi Ounadjela
• The future of data?by Pedram Navid
• A question is a start?by David Jayatillake
• Palantir Foundry: the data operating system that is not talked about enough?by Robert Kossendey
• Marissa Mayer - AI Insights and Walmart, Google & Yahoo

?? The Big Reveal of Atlan AI – 7 Days to Go!

A few weeks ago, we announced the launch of Atlan AI as the first-ever copilot for data teams. And it’s almost time for its big reveal. The launch was also covered in?Benn’s newsletter.

If you are interested in learning how AI has completely revolutionized the way data teams discover, understand, and work with data, come join the waitlist to explore Atlan AI and save your spot for the launch event next week?here.

P.S. Liked reading this edition of the newsletter? I would love it if you could take a moment and share it with your friends on social! If someone shared this with you, subscribe to upcoming issues?here.