DATA MESH AND DATA WAREHOUSE
Bill Inmon
Founder, Chairman, CEO, Best-Selling Author, University of Denver & Scalefree Advisory Board Member
DATA MESH AND DATA WAREHOUSE:
BUILDING ON A FOUNDATION
By W H Inmon
Every new generation in IT has a fundamental choice to make – do I build on the existing foundation or do I tear it down? Once upon a time it made sense to tear it down. When the computer first appeared using the automation afforded by the computer, the computer paved the way for tearing down things that were done manually. That destruction was a very constructive form of destruction.
But is tearing something down something that already exists always a good thing to do? The people at data mesh have inferred that with data mesh you don’t need a data warehouse. The people that promote data mesh think that at last we are freed from the tyranny of having to build a data warehouse.
Let’s examine this line of thought.
What is the benefit of building a data warehouse
So what happens when you build the data mesh on top of data that is not believable? You get an eternal uneasiness about the data. You never really know whether your data you are operating on is really believable. But if you build your data mesh on top of a data warehouse, you have confidence in your data
The question then really boils down to – do you want to have confidence in your results or not? If you don’t care about having confidence in your results then you don’t need a data warehouse when you build your data mesh. But if you care about having confidence in your results you need a data warehouse. It is as simple as that.
领英推荐
Data mesh is not a replacement or substitute for a data warehouse.
So what do you gain by having data mesh without having a data warehouse? If you go that direction, you don’t have to go through the awful and feared exercise of having to integrate your data
But the fact is that the price of having confidence in your data MANDATES that the data be integrated. You CANNOT have confidence in your data unless your data is integrated. This simple fact is non negotiable.
If data mesh were to offer itself as an extension of believability, then great things will happen. But as long as data mesh offers itself as an alternative to operating on believable data, then data mesh is doomed to go the way of other failures in our IT profession.
Building data mesh on top of an unvetted, unbelievable foundation is like the skyscraper in San Francisco that was not built on bedrock. You don’t want to be on Market Street or Chinatown when the building comes crashing down. Nothing good is going to happen then.
But this principle is not limited to data mesh. This is true of a whole bevy of other technologies that depend on operating on believable data. This principle is true for AI, ML, OLAP, the dimensional model, and a whole host of other technologies.
Once upon a time it may have been useful to destroy the existing foundation. But that day is no longer. It is time to build on the existing foundation.
Bill Inmon lives in Denver, Colorado with his wife and his Scotty dogs Jeb and Lena. Lena is younger than Jeb and runs faster than he can. But Jeb still can run fast for his dinner.
Senior Cloud Data Engineer at Future Processing
2 年My two cents (as a data engineer). From readings I did my understanding is that DWH and Data Mesh can co-exist simultaneously. Key element that points 'which path to choose' is a question 'where this data will be used'. DWH will remain as a key analytics/kpi/clean metrics solution since its capabilities to store and aggregate data. Data Mesh on the other hand could be used for robust/on-demand/real-time analytics where those who inject data into it are held accountable for quality of that data; this leads to not only tech stack change but also change in co-workers trust. Coming of Data Mesh could benefit both worlds by bringing division and unloading tasks that dwh simply can't handle/is not designed for. Only risk of this co-existence is danger of unecessery workforce. Data Mesh is a great opportunity for more technical staff to be nominated as a 'tech data steward' of their department.
Advisory Services | GCC Leadership | Digital Transformation | Data Privacy & Security | Advanced Analytics | AI/ML
2 年Correct! There is no shortcut to success.
EMEA Software & Data Architect Lead
2 年Agree ! And I think data confidence comes only after deeper data checks with SME
Solutions Architect, ArangoDB
2 年Love it!
Head of Data Science ? AI/ML Leadership ? Digital Transformation Strategist ? P&L Impact in Pharmaceuticals & Life Sciences ? Advanced Analytics & Predictive Modeling ? Data Governance ? LLM & Generative AI Innovation
2 年Bill Inmon Great article and you have simplied the thinking really well! I love it ! I respectfully have different point of view on couple of things: 1. The assumption that data in DW is always believable, is not the case in real world. Ideally that is how it should be. Readers should not assume by default that data in DW is always believeable. They need to verify and take remedial action to make it believable. I thought that disclaimer is important in your article. The lineage, quality got to be ensured as well. 2. Integration means different things to different people. In order to mesh to work in real world, data should be intergated/connected at the business domain and business process levels. This is important because if there is problem in downstream business process, we should be able to lineage back to upstream process. Unfortunately, This is not the case in real world. In order for Data mesh to work, all business domains should be integrated in addition to business process integration. Point here is that integration is needed for Data Mesh to work.