The Enterprise Data Architecture Evolution - From Data Silos to Data Mesh

Digital transformation and data driven enterprise have become the primary focus and pursuit for many organizations. Many industries which had the luxury of affordability to remain with low digitization in their business hitherto, are forced to reinvent themselves to stay relevant to changing customer preferences, new paradigms and compete with cost and performance efficiency.

While various technologies like AI/ML, AR & VR, block chain, multi-cloud, federated analytics, distributed computing, IoT and edge have critical role in powering this digital transformation, it is a current reality that organizations are still struggling to ingest, store, manage, govern the data produced by these technologies at scale and make use of the data at will.

Data driven organizations have made a conscious effort to commit themselves to data democratization. Every part of their organization is empowered with agility in access to data and the speed with which the decisions are being made leveraging it. This journey has been a long and arduous one. A compelling trend has emerged in the effort of these pioneering organizations.

Data Mesh. A new paradigm in how the organizations can leverage data. More importantly, how the data is being served to their internal organizations as a product.  

At a high level, data mesh is not a singular monolithic or discreet technology to implement. It is rather a methodology, an architectural concept and a novel approach to how internal organizations can acquire data and create a “quality data product” for other users and applications to make use of it, while leveraging it themselves.

Now, in order to understand the concept of data mesh and its horse power on agility, it is important to understand the journey the organizations have taken.

Data silos, rigid governance and tight controls of them served well for a long time. Mostly the data was transactional and the only way anyone have access to data was by a request to IT department. Depending on the intricacy and complexity of the request for data, it even took a year to identify the data sources, create data mart and provide access to other organizations within an enterprise. Innovating with data was never the primary objective. Reporting ruled the workloads on data.

When the paradigm changed around the value of data and how it was leveraged, organizations started to bring in more data. This included variety and in turn, volume. Complexities associated with ingesting and storing led the organizations to think beyond data warehouses for analytics. In came Hadoop. Storing became easy. However, complexities associated with quality, ownership, lineage, governance, security, sharing and serving the data increased by multitudes. Internal users had access to data “relatively” easier. But, lack of quality and context to the data (domain knowledge) compounded the challenges in leveraging the data.

The very foundational idea the organizations set to solve, “leveraging data for transformation” remained a quest. The single most contributing factor to it was the inability to serve high quality Data As A Service in a timely manner.

“Data Mesh”, the new enterprise data architecture plays a crucial and critical role in solving this major challenge. In the simplest of abstract view, a data mesh architecture is a change from:

• Storing and managing data in a centralized monolithic data architecture to a decentralized distributed architecture.
• Centralized IT ownership of data to domain level ownership (respective organizations within an enterprise)
• Data users being responsible to clean, cure and organize (quality of data) to data owners themselves (Remember the analogy, garbage in garbage out?)
• Delivering data in complex large files to a simple high-quality product.
• Serving the data in a time-consuming manual process to automated self-service that doesn’t compromise the security and governance of data.

Without making it too complex, an example of a data mesh in a manufacturing industry in a high level. (Since it is easier to follow) 

Finally,

• To be able to embrace Data Mesh architecture successfully, organizations need to restructure their data platforms, redefine the roles of data domain owners, and transition to treating their analytical data as a product.
• Data Mesh approach does not compromise any of the existing organizational investment in data management, wrangling or governance products. Rather, it is complimentary and maximizes the derivative value of these investments.
• Importance on quality at respective domain level and servicing the data in an automated way solves 90% of hardship involved in the effort of “deriving value out of data”.

In an era where companies compete on agility and efficiency to deliver products and hyper personalized experiences to win the heart and minds of customers, data mesh architecture delivers the compelling value proposition in the digital transformation journey.

Last, opinion which I have shared on this topic are my personal view.

#Datamesh #Bigdata #Datamanagement #Data-as-a-service