Building a successful Data Mesh – More than just a technology initiative

By: Nazia Shahrin

The Data Mesh architecture challenges the monolithic organizational empires that have been built to protect, curate and distribute data over the years; particularly the centralized information management teams. It recognizes that the data landscape is far too big and varied to be able to be managed centrally anymore, and require a network of teams who are able to take ownership, with technical capabilities that allow them to produce and consume in a seamless manner.

At this point, we need to take a minute and think about how this architecture will succeed when so many other well-thought out, expensive data strategies such as the warehouse and the lake have failed?

Zhamak Dehghani formalized this architectural-style in her article, “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” She described a new way of managing data in a complex, large organization,?suggesting that the next enterprise data platform architecture is in the convergence of Distributed Domain Driven Architecture, Self-serve Platform Design, and Product Thinking with Data.??

The data mesh is an architectural paradigm that exposes analytical data at scale; rapidly unlocking access to an ever-growing number of distributed domain data sets. The technology exists now to make that happen for an organizations data. You can produce the most accurate, complete, timely data from each distributed domain and put it together the way you want in your system. Compared to the struggles of finding your way around the ‘data swamp’, and the famous ‘garbage in garbage out’ problem of warehouses, this indeed sounds like a dream. But wait… data is never really just about the technology solution. Data Management, Governance, remediation… these all go beyond just the technology. This becomes about the overall capability, including processes and the people who now have to shift their mindset to a federated model of managing and owning data. A federated model creates an environment where both autonomous data domain teams and centralized functions collaborate to meet the needs of the whole organization. If a company truly wants to succeed in the implementation of their Data Mesh strategy, they cannot forget about what makes it what it is: the people, the processes and the culture.

Let’s begin with a short history lesson:

In order to understand some of the pre-requisites of a successful Data Mesh we must look back and explore what came before it - centralized keepers of Data like the Data Warehouses and Data Lakes.?I will use Banking as an example industry to illustrate both what came before and what must come after as everyone is familiar with day-to-day banking products and services.

Banks started with a simple purpose - they give people a secure place to store their assets.?As time passed they allowed people to deposit in one location and withdraw in another.?They began offering loans to people for significant purchases and later provided new products to allow people to invest their assets.?This simple set of products have stood the test of time and form the basis of all traditional banks. Banks structured themselves around these products - effectively forming multiple organizations inside a whole, each with their own processes, systems, cultures and so on.?

These organizations, let’s call them businesses lines, were all part of a single bank but were, to all intents and purposes, largely autonomous in all but a few areas.?At the heart of each of business lines was one or more core systems that were responsible for keeping track of customer transactions - the money they had deposited, the payments they had made, the loans they had taken out etc.?These systems “run the bank” and are the absolute truth of what the bank believes a customer holds at any one time. They have a well-deserved reputation of cast-iron solidity - handling millions, if not billions, of transactions every year and having done so, in many cases, for decades.

These systems are great at processing these transactions, it’s what they’re optimized for, but are normally very lacking when it comes to the ability to perform reporting and other data functions.?In the early days, Business intelligence was rudimentary, expensive, and siloed. File structures tended to be complex, diverse, and unique to each application, so that every attempt to use the data required custom analysis. Statistics was a largely academic tools used at scale by a few firms with close ties to research, there was almost no example of machine learning in commercial use those days.

The Rise of the Warehouse and centralized governance of Enterprise Data

A Data Warehouse is a database optimized for performing reporting operations on large volumes of data.?Logically and physically separate from core transaction systems, these databases are normally centralized across an enterprise and fed data, usually overnight, from core transactional systems.?There is a neatly structured segregation of duties here - core systems processing financial transactions and the data warehouse undertaking reporting for regulatory or operational purposes (e.g. marketing planning).??The teams that were formed around these data warehouses were very different to those in place to support the core systems.?Made up of data specialists (Database Administrators, Report Writers, Analysts), their focus was on ensuring the data was well governed, well-structured and communicated to other teams or executives usually in the form of reports or dashboards.

Over time, structure and strong governance mechanisms that were part and parcel of successful data warehouse initiatives actually started getting in the way of extracting value from the data.?There was a perfect storm - a massive growth in data volume and a meteoric rise in the demand for insights to answer complex questions.?Gone were the days of simple and static reporting and instead there was a need for real-time analytics, flexibility, sandbox experimentation areas and generally the need to support wider range of use-cases.

Companies began looking for alternative data solutions that could handle these dynamic requirements. This led to the large-scale adoption of Data Lakes - a technology more suited for the flexible requirements of today's data-based economy.?Data Lakes gave the ability to handle the same, or greater, data volumes with a key differentiator. There was no longer a strict need to keep data structured so allowing lakes to support multiple types of data more easily and to drive out value by pursuing more flexible data analysis. Report Writers and Analysts of the 20th century became the Data Scientists and Data Engineers of the 21st and embraced a period of unprecedented change in the data industry.

Teams surrounding these new systems, and the demand for skilled resources to run them, grew at a tremendous rate to the extent that many data groups began to match, or outweigh, the existing technology teams running critical systems.?High cost of data resources combined with the large teams required resulted in a high overall cost of running a data operation at scale.?The expectation was that benefits of the Data Lake would outweigh the costs to run it but more on that later…

?

So why do we now need a Data Mesh?

Looking back at the Data Warehouse and the Data Lake models, it is clear that little changed from an organizational structure perspective.?As before, disparate core transactional systems and teams continued to send their data on a regular basis to some form of centralized repository where a separate, also centralized, team would undertake data reporting and analytics functions.?As before, core teams concerned themselves with the ongoing running of the transactional systems whilst the data / analytics function focused entirely on reporting and extracting value from the data sent to them by core teams.?As transaction systems grew, so did the core teams and as a result centralized data teams also grew to handle the additional complexity and volume of incoming data.

Over time, the central data function, far removed from data creation, was at a disadvantage, unable to extract most value from the data. The central data teams were several steps removed from its actual creation and weren’t the experts in the actual subject.?They were experts in the data pipeline and the warehouse/lake processes but did not understand the product and transaction data they were trying to derive value out of. For example, central data teams were unlikely to fully understand the myriad of data resulting from a complex commercial loan purchase because they do not live and breathe that process. They have little connection to the complexities of the business or the products, resulting in a limited understanding of the complex data models of the core transaction systems. The additional complexity also lies in the legacy platforms and manual intervention in the process, resulting in quality, completeness and integrity issues in the data flowing into the central structures.

As use cases grew in scale and complexity, this lack of subject-matter, or domain, knowledge became a limiting factor. Sheer scale, complexity and richness of the data has pushed the centralized data model to its limits

This is where a Data Mesh comes in.?A Data Mesh is effectively an implementation of a distributed architecture where structured logical groupings, known as domains, become providers of data to the enterprise and form a federated data platform, or ecosystem, that allows consumers to easily draw data from one or more domains as required.?

This approach brings with it various benefits not the least of which being the fact that it is the domains, the experts themselves, who make the data available for consumption by others - the people who understand the data the best ensure it is consumed and used to derive real business value by its users.

Continuing with the Banking example, the domains in a Data Mesh architecture would most likely be product-related - Credit, Deposits, Investments etc. although, as with everything in this model, the ability to change and adapt over time is key so these structures should not be set in stone.?These domains will vary from company to company and defining them is as much art as it is science.

?

We want to build the Mesh….how do we make the most of it?

A federated model such as a data mesh requires a higher level of maturity in the organizations data management culture as, by its nature, it represents a very different way of working both in terms of interacting with other groups and with the data itself.?A careful balancing act between competing pressures is required in order to reap the full range of benefits from the architecture.?In many ways, teams must go on a similar journey to that encountered by development teams with the adoption of agile, DevOps and API-centric thinking.

Large organizations such as banks in general have been process centric on the business side and application centric on the technology side, resulting in a mature process and system culture, but focusing less on maturing the capabilities needed on both sides for the data. Data is usually a by-product of the process, and now needs to be treated as a product itself.?Data leaders have shown up in the highest levels of the organizations but have often struggled to penetrate the daily workings of the core teams with data management values.

To move towards a successful data mesh, there are a few components to it that need to be implemented alongside the technology. These components require the organization to evolving the culture and maturing data management.

Evolving the culture

To build the right culture for the data mesh, leaders must support and empower an organization-wide cultural evolution. This is one of the difficult accomplishments for leadership, especially in successful, large companies. The reason for this is that employees are comfortable in their ways, and it has made the company profitable. A distributed data management model will require existing employees to step back from their old processes to a completely new mindset. The following evolution of ‘ways of working’ will need to happen to make the Data Mesh successful.

Data as a Product – ‘Think like a product team - be one with the data’

An organization must be able to treat its data like a product - something that has to be cared for over time and evolved. Imagine walking into a store and picking up a bottle of juice. Standards and guidelines have said that every bottle of juice produced must have its ingredients listed and its calories stated. The process the juice goes through also ensures it meets quality standards. These are just expected norms now. If you don’t display the ingredients and don’t ensure quality – you will not make your way into the shelf of a proper store. If you translate this to data, it means the data produced by the domain, must have metadata describing their data product and for it to be of a high quality so consumers can be confident that it will serve their needs.

Many domains have a head-start here through the wide adoption of API-centric thinking where product thinking - the idea of treating the APIs as products themselves, making them high quality, easy to discover, easy to work with, reliable and evolvable - is already established.?A similar set of concerns are important when considering how best to succeed with a Data Mesh.?In a Data Mesh architecture, each domain must continue to ensure its product, its data, is of value to its customers: the rest of the organization outside the domain - the data scientists, analysts, engineers and process teams who will be using the product to build their own solutions and meet their own particular requirements.?Remember - part of the reason for the emergence of this architectural style is the desire to have the actual subject matter experts directly involved in the ongoing life of the data to bring their skills to bear.?

Domain teams have to start thinking differently about the data – they are now producers of a product and that product has to have certain capabilities and traits for its internal and external consumers. As Dehghani states in her article, “For a distributed data platform to be successful, domain data teams must apply product thinking with similar rigor to the datasets that they provide; considering their data assets as their products and the rest of the organization's data scientists, ML and data engineers as their customers.”

This will be a major mindset shift for banks where the customer is only seen as the client, and not internal consumers of data from the process.

Decentralize Trust – ‘Be comfortable with federating trust and governance’

A data mesh federates ownership of the data and this puts additional challenges on the way data is managed and maintained.?We can no longer rely on centralized controls covering a single monolithic data store, instead trust is now decentralized and the organization must be comfortable with that fact.?A mindset shift is required and to support this and each domain must have the appropriate skills and controls in place to allow it to actively govern its data.?This therefore requires each domain to build up its level of maturity in the practice of data management – establishing roles and processes to ensure the data provided by the domain, its product in the eyes of data consumers, is of the highest quality.?

For many domains this is simply an extension or formalization of activities that already exist - the data within the systems having always been managed, albeit with more of a transactional lens.?Domain data management teams will be accountable for the functions often performed by centralized data management office groups and the like - focusing on data accuracy, completeness, consistency and timeliness to make the data as trustworthy and useful as possible.

To make the most of a Data Mesh architecture it is critical for domains to look beyond what was previously their sphere of responsibility.?For many, once a piece of data was successfully delivered to a consumer by a system, it was no longer of any real concern - other teams, systems or processes could modify, enhance or otherwise manipulate the data as needed.?This will be a major shift in mindset when the output becomes a product. Just like the core teams have maintained the systems and processes in the past, they now have to maintain the data as a product and this will have an impact on their mandates, their day-to-day activities and will create a learning curve.

A fundamental shift required is that an organization must learn to trust multiple, federated domains rather than a single, centralized function.?Of course, these domains are already trusted to provide particular capabilities but being trusted to provide data as part of a mesh architecture is a very different situation.

This goes hand in hand with shedding the perceived notion that it is better to go to one place for lots of data rather than lots of places each for a small piece of data.?Instead, any organization looking to implement a data mesh must be comfortable with assembling data from multiple sources - taking only the data it needs from each domain and combining it to generate insights sought.?This approach allows consumers to take the very best data right from the source - giving them the finest quality ingredients to work with.

Ease of consumption is another area where a cultural shift is likely to be required.?Historically, data has often been made available in native and local structures and placed a burden on consumers to interpret.?Instead, care must be taken to make the data as easy to consume as possible by harmonizing within the domain wherever possible.?This may require efforts to standardize the data within systems or to apply translation logic prior to publishing.?The domain taking on effort provides a higher quality, usable product for the consumers rather than raw material, the consumer has to standardize.

It is a similar story across the domains.?One of the key features of the Data Mesh architecture is that consumers are likely to require data from multiple domains and therefore anything that can be done to increase the ease by which this occurs will certainly provide benefits.?This therefore highlights the need for domains to have a high degree of interoperability – standardizing to allow consumers to more easily traverse domains.

Balance Risk & Control – ‘Find the balance for just the right amount of control’

Volume, variety and velocity of data and our needs is already challenging our ability to manage risk and control. Risk and controls functions are catching up to innovations stemming from data. As an organization moves to a domain model, they need to reassess their risk appetite and control processes. Is it even feasible now to control every piece of data that flows through an organization centrally, or do we set overarching policies and practices that ensure that the domain teams can move the organization forward without jumping through a million hurdles? This component ties closely with decentralizing trust and asks us to get beyond the ‘one size fits all’ approach for risk and control. There may be domains that are heavily controlled, and others that might not require as much scrutiny. The right balance will be key.

In banks, as in most organizations, the client domain (responsible for storing information about customers) will always need to be carefully monitored domain and must have the right domain-level controls to ensure a client’s privacy. This is particularly pertinent given the prevalence of client data hacks or accidental leakage.?Other domains that are less regulated may not require the same level maturity although, of course, they may opt for a higher level of control than strictly necessary. The domain model needs to rely on the SME’s of each domain to determine the risk appetite of that domain, and build the right infrastructure around it. This will again challenge some of the centralized risk and control teams into federating out some of their responsibilities to the domains, while still maintaining a central oversight function.

?

Federated Data Management - 'One to Many'

Data Analytics is the ‘cool kid’ on the block, while data management is the kid asking you to follow the rules. It’s no surprise that most of the investment to grow the data practice in organizations has been about bringing in the technical capabilities for analytics and maturing data management has been pushed to the side. This needs to change for the Data Mesh to work.?The Data Mesh needs Federated Data Management and Data Governance to be mature and be a focus for each domain at the local level.

Data Governance – ‘Global and Local Governance is the way to go’

A successful Data Mesh necessitates governance that allow domains to operate with a high degree of independence but also does not lose sight of the fact that interdependence between the domains is equally critical.?Picture an environment in which every domain had complete autonomy with no attempt made to coordinate or drive consistency across areas….it would quickly become like the proverbial Wild West, a place where every activity is disorganized.?But similarly, an authoritarian approach forbidding domains to act independently would massively constrain the value of the architecture.?Instead, follow a federated model of governance where domains are empowered to make decisions that affect themselves but also allow a higher order of authority, a collective of the domain product owners perhaps, to make decisions that affect all domains.?This allows domains the freedom to evolve in their own way and at their own pace whilst still ensuring interoperability remains at the forefront of the architecture.?Organization need to be prepared for this balance to change over time as the domains mature and as consumers become more ambitious with their requirements.

Data Governance capabilities ensure the data is as good as it can be within reason. Even after all the quality checks of produce, there will always be an apple that is rotten on the inside. This is not about the special cases, but more for the general. Dehghani points out in her article that no one will use a product they can’t trust. The local domain level Data Governance capabilities will ensure the product is trustworthy with quality, integrity, timeliness and others in check.

In banking organizations, it is likely that Payments groups will be in the vanguard of adopting governance models of this nature given their involvement in broader payments ecosystems – a heavily standards led area where interoperability and quality / integrity are absolutely critical.

Security & Privacy – “Re-think security and privacy from system level approach”

The Domain team must take a zero-trust approach to the data as they publish it for others to consume. Confidential and sensitive data must be protected from the beginning to reduce copying and movement that is likely to occur after that data has been consumed by the domain. Instead of controlling the data at every point, create a zero trust approach from the initial distribution and create an access based approach for any application or process that may require this data.

In practice, this means that consumers will no longer receive a large dataset and discard what is unnecessary but will instead only be provided to the limited set of data they have a need for and permission to.?This will necessitate additional processes, checks and balances to ensure anyone consuming data is doing so appropriately.

For large financial services companies, this change in thinking opens greater possibilities for collaboration with external organizations, be they fintechs or other partners in an ecosystem or network such as loyalty reward schemes.?With the global trend towards Open Banking, these are shifts that many organizations are at least considering but the emergence of the data mesh architectural style provides an opportunity to accelerate this process.

Metadata & Standardization – “Make it easy to find and consume the data products”

One of the biggest challenges in large organizations is often that data is difficult to find - locked away in a system somewhere - or too easy to find - multiple copies proliferated across the organization with varying degrees of quality.?This paradox can sometimes be attributed to the fact that the demand for key data is so strong that when it finally is made available there is a sudden, ungoverned, rush to consume and share. Dehghani points out in her article that “A data product must be easily discoverable. A common implementation is to have a registry, a data catalogue, of all available data products with their meta information such as their owners, source of origin, lineage, sample datasets, etc.”

A logical and well-defined set of domains inherently makes it easier to identify where data should be available – directories, glossaries, search tools and documentation wikis will all help make this so.?The domain structure must make sense to the organization and be grouped in such a way so as to allow consumers to easily understand its arrangement and easily navigate to the data they require.?Larger domains may be split into smaller sub-domains if required but care must be taken to not be overly abstract in this area as this can act as a barrier to comprehension for consumers.?

This is where a strong metadata management capability will make a significant difference in how teams find and consume data. In a domain based approach, a good data catalog that makes data easy to discover, understand and consume will make the most impact. This means that individual domains need to maintain their business, technical and operational data, and there has to be a consistency of infrastructure for metadata across all the domains (otherwise consumers will be overwhelmed by different tools and processes). For example, the payments domain needs to ensure that their business terms, technical field level details and simple operational data such as frequency/latency are all available to the consumer to be able to work with the data. It sounds fairly simple, but this data historically was never documented or managed this way, and will require initial and ongoing investment from business and IT teams.

The experience of the consumers will play a major role in the success of the data mesh, they will be shifting from a centralized set of individuals servicing their data requests to a self-serve approach. Imagine if the self-service check outs at the grocery were significantly harder than going to the cashier, why would the customer use it? A similar analogy applies here, if it is harder to discover and understand data from the mesh, even if the quality is better the consumer will go to the old architecture.?

In addition to metadata, standardization will enable ease of consumption. Dehghani notes that “Quality products require no consumer hand holding to be used: they can be independently discovered, understood and consumed.” This is another area where a cultural shift is likely to be required.?Historically, data has often been made available in native structures and so places a burden on consumers to interpret.?Instead, care must be taken to make the data as easy to consume as possible by harmonizing within the domain wherever possible.?This may require efforts to standardize the data within systems or to apply translation logic prior to publishing - the domain taking on effort that would otherwise fall to the consumers.?This is one of the most critical factors of success for the Data Mesh – it must be as easy as possible for consumers to do ‘the right thing’ for the architecture if it is to succeed.

It is a similar story across the domains.?One of the key features of the Data Mesh architecture is that consumers are likely to require data from multiple domains and therefore anything that can be done to increase the ease by which this occurs will certainly provide benefits. Dehghani addresses this, “The key for an effective correlation of data across domains is following certain standards and harmonization rules.”?This therefore highlights the need for domains to have a high degree of interoperability – standardizing on how consumers should interact with the domains and driving consistency to make it easier for them to access composite data from across the various domains.?If the way a consumer interacts with one domain varies from another then that is one more barrier for them to overcome and so increases the likelihood that they will look for an easy option.?The expectation has to be high for the producers and publishers of the data and lower for the consumers. This will ensure scalability and adoption of the domains.

?

Finally…Be prepared for ambiguity and some difficult discussions…

When implementing a Data Mesh architecture, or anything Domain-related for that matter, an organization should be prepared to have some difficult discussions and make some tough decisions.?Domain boundaries can sometimes be blurry and there is likely to be discussion around which data are best provisioned from where.?Similarly, domain boundaries will almost certainly not match existing team or organizational structures and nor will they necessarily map neatly onto systems or existing APIs (etc).?

And this leads onto perhaps the toughest decision of them all - what to do with systems, APIs and so on that do not fit the new Data Mesh architecture: existing APIs, data repositories such as lakes and any other technologies that are unable to form part of the mesh??These systems represent a considerable amount of investment, effort and resources and for many they are a sacred cow that cannot be sacrificed at the altar of a new architecture.?

It is at this point where discussions will gravitate to some form of hybrid architecture or other compromise in order to preserve some of the existing data platform.?There is no easy answer but it is critical that senior leadership from across the organization be fully engaged in any data mesh initiative for the simple fact that they are likely to be required to arbitrate on key decisions.

?

Conclusion

The Data Mesh architectural style is a relatively new architectural style that will allow organizations to move beyond the monolithic architectures that favor data lakes and data warehouses.?However, in order to truly succeed with this model it is imperative that organizations understand the implications inherent in the style and make the appropriate changes holistically to make it successful.?

With all the attention currently focused on Data Mesh architectures there is a chance that organizations will adopt it as a technology strategy without considering the fact that it requires changes beyond technology if it is to be successful.?It has happened before with other architectural styles or technology capabilities and it is for this reason that this article was written – to raise awareness that a Data Mesh architecture has real potential in large, fragmented organizations such as banks but it cannot simply be investment in new technology.

References:

Dehghani, Zhamak (2019, May 20).?How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. martinFowler.com. https://martinfowler.com/articles/data-monolith-to-mesh.html