Evolution of Data Mesh

Why is Data Strategy so important?

According to study done by McKinsey here are some startling facts about Digital Transformation Failures. What these numbers say is that the Digital transformation pivots on data quality, quantity, timeliness and accuracy.

Data is amazing, and is the most critical and the most significant component of knowledge. It embodies both time and cost sensitive information for our ecosystem's sustainability and growth in today's technology intensive landscapes. Anything living, breathing, transacting, or thriving around us is driven by the power of data, be it through a set of rules, cognitive mechanism, analytics, insights, decisioning, or for dynamic adaptation.

On the horizon across industries, we can see a massive and collective push towards forming digitally transformed modern landscapes. Unfortunately, we often witness a high rate of failures there as well mainly due to their north star architectures missing data centric thought leadership, strategic data architectures (e.g. Data Microservices), and guardrails around data organization, ownership, federated security, transmission, and distribution. Without the right strategy in place any supporting volume, variety, or velocity won't help us monetizing on data. For our awareness, this is not just a one time effort, but a continuous journey to sustain, achieve, and augment a set of standards and KPIs on the data landscape.

What is "Good Data" and "Bad Data"?

Good Data is like a "currency" not frequently reprinted by the organization and with "right and tight" ownership, governance, and authority. It enables effective distribution and frictionless utilization. The intent is not to restrict data, but to ensure that the quality, integrity and sensitivity is not degraded as it traverses. This makes it highly reliable, dependable, and traceable across the organization without intentional or unintentional impurities being layered over. The benefits of good data to the business are:

1. Easily Accessible - Can be accessed with minimal friction
2. Easily Transferrable - Can be federated for utilization
3. Must be Reusable - Treated as a product, not like oil
4. Easily Integrable - Can be flexible with various interfaces
5. Domain Driven - Must focus on business, not monoliths
6. Supports Innovation - Ability to experiment and adapt

Bad Data is essentially the data which contributes to all the odds of qualities given above. Let's say we have a Customer table in CRM App with the name customer_id and data type as NUMBER(9) originally defined. Per the need of business flow, the value in this field is relayed to a downstream order management app (OM). The OM app then converts customer_id into a different data type and is given a new name. The same happens within cascading downstream apps such as Usage, Billing, AR, etc that process transactional data against the customer entity and translates data pertaining to customer, products, and services into revenue.

What has been observed is delivery teams take approaches to help businesses improve time to market by taking risk endeavors. As different downstream apps do not foresee the clutter this creates as a consequence by creating a variety of names as customer_number, account_number, and customer with data types as NUMBER(9,2), NUMBER(14), and VARCHAR(12) respectively. This risks integrity, quality, traceability of data, and makes it extremely challenging with higher effort needs at every point due to potential misleading information, further snowballing into an egregious problem and challenges our transformation or modernization journeys. The few difficulties caused by bad data can be:

1. Operational Overheads - Frequent failures, surprises, and resource crunch
2. Infrastructure Overheads - Needs escalated back-end system capacities
3. Processing Overheads - Multiple formats, scattered re/ conversion needs
4. Slows Time to Market - No domain orientation, silo implementations
5. Inflexible Integration - Always needs additional adaptation layers, cost, and time
6. Reverse Engineering Challenges - Intertwined monoliths, more friction, time and cost
7. Inconsistent Reporting - Users and Business receives misleading information

Modern Data Landscape

For data to be purpose driven, it needs to be designed strategically as a product, traversing within or across ecosystems reaching living and non-living entities. The ecosystems can be our own mind, nature, the universe, a personal computer, or varieties of businesses and social platforms, interchangeably playing the roles as producers and consumers. To get the targeted ROI from a specific product, entrusting ownership, governance, control, and a strategic roadmap with short and long term goals is a must.

In modern industry landscape, millions of transactions bombard the systems of interactions, engagements, and services every second. Secure and effective distribution of these transactions, processing, and massive monitoring and logging activities provide enough information about the participating entities (businesses, customers, machines, and things), and how they impact an ecosystem. If we study these patterns, it opens up a wide range of opportunities and possibilities that can harness and accelerate business growth by making it more vigilant and intelligent. Perspectives and features embedded in the data can be a holistic voice of the customers, users, or the system itself, working as the ultimate driver and differentiator in the market.

Modern data landscapes interchangeably treat data as the "currency" or "product" to create business agility, domain orientation, and customer centricity for sustainable and predictable top line growth trajectory. The organizations which have readily adopted such a model, are successfully building a dynamic environment with sharp customer-centric products and drive hyper-personalized services for the best differentiable customer experience.

The image Fig 2 below, it shows how analytics data landscapes are evolving across industries. Unless we realize the mess early, it may already be creating your next monolith and inherently hitting hard on our cost of ownerships and time to market. In the image the first stage shows lack of data and inability to build many features. The second stage shows a data hungry landscape with flooding of data which is when the mess gets worse. And, in the third stage it shows how to normalize the landscape, organize, and make it more business focused by bringing domain orientation. The idea here is not to discredit Data Lakes or Lake House formation concepts but to approach them right to declutter business by using a Data Mesh approach.

What is a "Data Mess"?

Over the last couple of decades, most organizations across industries have been aggressively trying to dig into data to monetize it for the betterment of their top-line and bottom-line. With that intent in mind, organizations collected massive amounts of data supporting velocity, variety, and volume. They got engaged in building supportive infrastructure and aligned resources. Today, even the smallest of businesses cannot afford not to use the value in data and have adopted methods like simple spreadsheets to record data and make decisions based on the patterns observed. There goes a fair amount of cognition backed by our experiences over the years, layered over the logged data and mathematical formulas for business sustainability and growth.

For businesses turning over at scale with high volume of transactions, we need broader capabilities and system capacities. So applying methods like spreadsheet recording becomes insufficient. Looking at any large business, we can find hundreds of data producers and even much higher numbers of consumers in line with many dynamic attributes that can give rise to tangible outcomes. Because of these engrained dynamics in large businesses, it is very necessary that we collect as much data from all possible sources and use them towards our decision support system. This will ensure no missing dimensions around patterns, expressions, or utterances. To give data the targeted purpose, it needs to flow across and between various departments being securely guard railed, so that the various dimensions do not go out of proportion, and prevents an unimaginable "Data Mess".

What is a "Data Mesh"?

With the high data velocity, volume and variety, large businesses constantly struggle for higher capacities to process (using VMs, Containers, Serverless, etc.), to transmit data using network (Cloud Infrastructure, CSPs, Private Networks, Direct Connections, etc.), to distribute (using Pub/ Sub, Batch, File Transfers, Streaming, etc.), and to store data (using NFS, SAN, SQL relational/ NoSQL, Document, and Graph DBs on Data lakes, DW, lake house, etc.). To gain better control over this nuclear explosion of repeatedly reprinted data, which is needed to support evolving business flow complexities, we need better data and data driven solutions. However, the traditional approach of reprinting data is not apt, and has complicated this landscape even further, making them vendor sticky, black-boxed, complex, challenging for transformation and modernization journeys causing eventual failures and resources wasted.

While these unintended and unfortunate outcomes demand for better, more modern, and rationalized data architectures in place, data intensive organizations must break the existing monolithic centralized data architecture. The new architecture needs to share over a well governed data fabric for authenticated and authorized access. This thought process is in the right direction to define fine grained ownership within a subscription model for the consumers. This approach contributes to higher mobility, security, better governance, for purposeful data from/ and for business domains. That's how the data as a product can trigger agile processes which are event driven, decentralized, and federating business functions for a holistic data strategy. There are several benefits to the business from such a landscape in the form of higher availability, resiliency, responsiveness, performance efficiency, and cost-effective platforms. Below image summarizes what to expect from different type of data landscapes. It shows what most of the organizations current state are, and what their expectation in those phases with or without Data Mesh.

Conclusion

When it comes to implementing "Data Mesh", there is "No One Size Fit All". We know that many organizations are dealing with their "Data Mess" at different stages in the journey to attain moksha. If the implementation is not thought thru strategically, it can become highly counterproductive for any organization. A better assessment of the landscape, choosing the right technology partners, and building a step by step roadmap to reach north star will be critical for our success.

Here I am sharing my personal experience with a few clients who are brainstorming and are already in the journey. My strong recommendation would be to not separate "Data Fabric" and Data Mesh" as different concepts, rather use them as a complementing actors. If you are familiar with Orchestration fabric, please engage such resources with strong data modeling expertise to create the right domain & subdomain orientation. When we treat data as product make sure you also think how to maximize the revenue through the product and boost your ROI. Keeping the options open, flexible, and adopting a similar Microservices approach as you do with Digital with domain orientation and gradually strangling data monoliths will be a seamless and more robust approach.

I am sharing here the references of few successful implementations by organizations following the guardrails and principles we described above.

Please do not forget to share your thoughts, comments, and feedback. Happy reading!

Thank You!!

References:

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes Beyond the Data Lake - https://youtu.be/eiUhV56uVUc

Breaking Analysis: How JPMC is Implementing a Data Mesh Architecture on the AWS Cloud - https://youtu.be/s8cboADmVtA

Data Mesh: Lessons from the trenches – Sina Jahan and Storm Heidinger – XConf North America 2021 - https://youtu.be/CwC3kuShX6U