Decentralizing Data: From Data Monolith to Data Mesh with Zhamak Dehghani

Key Takeaways:

• Learn what inspired Zhamak to identify an inflection point of data management and create Data Mesh—one of the most exciting shifts in how we manage data at scale
• Understand why traditional data architecture models are failing
• How you can apply product thinking principles to data management to harvest your data’s full potential
• Why the convergence of product thinking, data management, and distributed systems development are reframing how we create platforms and products of the future.
• Formulate the first steps of your data management strategy
• Hear how your teams can start to see both internal and external customers of the data they create, to improve data literacy, high quality insights and better decision-making for the products you’re creating

Early Values

Zhamak has always believed in distribution of responsibility and decentralization of ownership. She finds that these design principles are more compatible with real life. Colleagues taught her the Unix philosophy early in her career which now forms the basis of her data management approach. “They taught me those wonderful ideas to build systems and programs that do one thing and one thing really well. But most importantly they work together really well,” Zhamak says. “‘Simple is beautiful and beauty is the truth’… Reduce systems to their simple principles; then together can emerge complex behaviors.” She saw an opportunity to bring the UNIX principles to data. [Listen from 3:25]

Challenging Assumptions

It often takes someone new to a system to point out obvious flaws to long-time practitioners. Zhamak says that when she came into the world of big data, she was agnostic to the accepted assumptions, so she felt free to challenge them and conceive a different paradigm. For some reason when it comes to data, people eschew UNIX principles and see it as something to be centralized. Unsurprisingly, a data lake becomes monolithic and departments become siloed. Reimagining the world of data requires a new language, she points out: “The moment you need to imagine something different you need to use a very different language.” Instead of seeing data as an asset – which you want to hoard and get more of – Zhamak advocates that data can be seen as a product which should be used to serve internal and external customers. Barry adds that the idea of the single source of golden data makes companies unable to move as they get bigger. [Listen from 10:20]

“The moment you need to imagine something different you need to use a very different language…” – Zhamak Dehghani

Click To Tweet

Move to Product Thinking

Barry comments that the shift towards product thinking started with Amazon. Their monolithic database was preventing them from scaling. “They realized that they needed to create these smaller, more autonomous units that had the capabilities to build things just like product teams. This is where this notion started to emerge from changing the organizational design… both technically and just how teams would work together,” Barry says. In this new way of working, teams could experiment and own outcomes. They could make small, quick changes and see the effects. [Listen from 18:50]

What is Data Mesh?

Instead of trying to fit data into a mold, Zhamak feels that its dynamism should be embraced. “Create an architecture and ownership of data that starts with the assumption that data can be useful and shareable and trustworthy right at the point of origin; and then allow for different domains and different aggregations, different projections to get created as a mesh picture,” she posits. She explains how this new view of data impacts ways of working and the type of platform a company would create. The four principles of the Data Mesh philosophy are, “domain ownership of the data; data as a product; self serve data platform to enable autonomous teams; and a federated computational governance to balance the interoperability of a decentralized world with the trust and security built in.” [Listen from 23:55]

“Two areas that there is just so much space for innovation and magic… is the product thinking and bringing product development minds to our technical capabilities and platforms and data; and the other one is really organizational change…” – Zhamak Dehghani

Click To Tweet

Thinking Big and Starting Small

‘Think big. Start small. Move fast.’ Start with a big vision about where you want to go, Zhamak advises. Next, ask yourself where you can start. She says that you should start with just one or two use cases. Building out these small solutions and testing them will help you build the foundation to go further. Work backwards and see what data you need then seek it out. Then figure out the capabilities you need to build your first generation platform. Build the platform. Move to the next use case and rinse and repeat. “Even though you’re taking the small steps and moving fast you’re still moving towards that big vision,” she says. [Listen from 33:10]

Empower People To Use Data

Barry uses Lipton as an example to illustrate how decentralized data can help a company innovate. He and Zhamak agree that data should be served as a product available to the rest of the company so that they can combine, explore and build new data sets. New technology allows you to synthesize data to find gaps that you can fill with new offerings. Zhamak urges leaders to remove the friction: empower the people closest to the data to experiment. [Listen from 42:10]

Resources

Zhamak Dehghani on LinkedIn | Twitter

Data Monolith to Data Mesh article

Data Mesh Principles article

__________________________________________________________________________

Unlearn Podcast

Keep Learning Forward with Amy Farrow, CIO at Lyft Inc.

Amy is a collaborator, team builder, and problem solver at heart, traits that she brought to her former roles at Salesforce and Twitter, and which she continues to champion as the current Chief Information Officer of Lyft. We talk about the value of reflection and how to leverage learning to build a better organization, including what not to do in times of challenge or crisis

Leading an Engineering Culture Transformation with Chris Goddard, CTO at G-Research

Chris has been with G-Research, a leading quantitative research and technology company in the algorithmic investment space, for almost 20 years. On this show, Chris shares a few or his experiences throughout his career. From being thrown into a job he didn't sign up for to leading a cultural change in their company. It was a very refreshing conversation and I highly recommend you listen to the full podcast!

Want to absorb these kinds of ideas on the go and hear from industry leaders?

Listen and Subscribe to the Unlearn Podcast at

Apple Podcasts | Stitcher | Google Play | Google Podcasts | Spotify | iHeartRadio

__________________________________________________________________________

Events

Watch the recording of Harley-Davidson LIVE Innovation Strategy Session with Barry O'Reilly and Diana Kander

Harley-Davidson is not only an iconic brand, but also a very popular one in the B2C world. However, it is well known that they are having problems in the B2B world after being unceremoniously dropped off of the Fortune 500 biggest companies list. How could this have happened? Well, we have some ideas.

LIVE Thurs. March 18th 10am PT - Machine Unlearning with Bill Higgins, Director of core AI and data plane technology at IBM

The machine learning algorithm creates its own programmers that can perform computing tasks far beyond the capabilities of traditional programming, such as understanding human language or identifying particular objects in complex pictures or predicting demand at a hyper-localized level in a supply chain system.

While the machines learn to program, humans must unlearn how they have programmed to create better solutions together and exploit the capabilities of both.

In this livestream, Barry O'Reilly, Business Advisor, Entrepreneur, and Author, and Bill Higgins, Director of core AI and data plane technology at IBM, address some of the non-obvious problems and solutions in this new era of machine learning, so that leaders can be more ready to perform and adapt in this new era.

Save the date! March 23rd - Chapter Two with Laurie Ruettiemann

Here's another addition to the Chapter Two series where I'll be interviewing Laurie Ruettiemann, author of "Betting On You: How To Put Yourself First And (Finally) Take Control Of Your Career".

On Chapter Two we livestream with leading authors about their new books under three constraints;

• Author shares the BIG idea central to the book
• Author picks one chapter to do a deep-dive on
• 25 minutes max

Grab your copy of Laurie's book and click here to get notified FIRST when we go live. This is gonna be an exciting discussion you shouldn't miss!