Data Mesh vs Data Fabric

Article written by?Pedro Bonillo, ML and Big Data consultant at?INNOVANT

The democratization of big data and the scalability of big data architecture are aspects of utmost importance for companies that currently prioritize their decisions according to their data. Is important to understand the concepts of Data Mesh and Data Fabric correctly to make the right decisions.?

Surely your company's head of big data has already been evaluating the capabilities of snowflake or looker and looking for ways to evangelize teams about big data, and your CTO has surely invested time and money in building a team of big data engineers and big data scientists . However, even in your company, you probably do not have the versatility and maintainability necessary to respond quickly and adequately to the? data needs of other business areas.

In order to democratize data, scaling the data architecture and efficiently responding to business requirements, the concept of data mesh arises.

Data mesh architecture

As first defined by Zhamak Dehghani, a consultant at ThoughtWorks and the original architect of the term, a Data Mesh is a type of data platform architecture that embraces ubiquity (the ability to be present everywhere) of the company’s data, leveraging a domain-oriented, self-service design. Borrowing Eric Evans' domain-based design theory, a flexible and scalable software development paradigm that matches the structure and language of your code to its corresponding business domain.

This then consists of seeing the data as a product and consuming it through microservices according to the domain, all of this connected through an interoperability layer with the same syntax and standards, using the data lake as the single point of access to the data but exposing the data as microservices depending on the domain.

Privacy and data protection regulations, such as the European Union's General Data Protection Regulation (GDPR), have been the main drivers of data governance initiatives. Data governance applications have had to expand their management from a technical approach (master data management, data catalogues, data quality, etc.) to include data privacy, protection and sovereignty.?

Organizations have a growing appetite to harness their data for business advantage, whether through internal collaboration, cross-ecosystem data sharing, direct marketing, or as the basis for Business Intelligence-driven business decision-making. As they do so, organizations must be careful to maintain the trust of employees, partners, and customers in their approach to leveraging data (and data-driven technology).

The emerging data fabric design concept can be a strong solution to ever-present data management challenges such as high-cost, low-value data integration cycles, frequent maintenance of previous integrations, increasing demand for real-time and event-driven data exchange, etc.

Data fabric

Garner Group defines data fabric as a design concept that serves as an integrated layer of data and connection processes. Data fabric uses continuous analytics on existing, discoverable, and inferred metadata assets to support the design, deployment, and use of integrated, reusable data across all environments, including hybrid and multi-cloud platforms.

Data Fabric enables you to:?

1. Connect to any data source through pre-packaged components and connectors, thereby eliminating the need for coding.
2. Provide data ingestion and integration capabilities between data sources and applications.
3. Support batch, real-time Big Data use cases.
4. Manage multiple environments, such as Hybrid Cloud and Multi Cloud, both as a data source and as a data consumer.
5. Provide quality data, enable data preparation and data governance capabilities, enhanced by machine learning.
6. Allow the exchange of data with internal and external agents through API.

Data fabric vs Data Mesh

According to Yuhanna from Forrester, the key difference between the Data Mesh approach and the Data Fabric approach is in how the APIs are accessed.

?“A Data Mesh is basically an API-driven solution for developers, unlike a Data Fabric which is the opposite of a Data Mesh, where you write code for the APIs to interact. On the other hand, Data Fabric is codeless, meaning that API integration happens within the fabric without directly leveraging it, unlike Data Mesh.”

For James Serra, Data Platform Architecture Lead at EY (Earnst and Young) and previously Big Data and Data Warehousing Solutions Architect at Microsoft, the difference between the two approaches lies in user access.

“A data fabric is focused on technology, while a data mesh is focused on organizational change. Data Mesh is more about people and process than architecture, while Data Fabric is an architectural approach that addresses the complexity of data and metadata in an intelligent way that works well together.”

Finally both Data Mesh and Data Fabric have a seat at the big data table. When looking for architectural concepts and architectures to support your big data projects, it all comes down to finding what works best for your particular needs.