Data Mesh

For many companies Data Mesh is not an option. It’s too expensive and too long. And they don’t have the departmental skills in data engineering or data modelling. Only corporate giants like HSBC, Siemens and Shell have such departmental skills. They operate in like a hundred countries so why would they want to centralise anything. Of course they should decentralise. But most companies and organisations are not like that. They work in one building. They don’t have thousands of IT staff, but only forty or sixty. They only have a small development team, probably five to ten people. Of course you would want to centralise things. You want to build a data mart, like a few fact tables with their dimensions. You don’t even have the options to put in a middle layer (silver layer) like Data Vault or EDW. It is too expensive and too long.

But then we have the mid size companies. Companies who can afford to have a silver layer. Companies with annual IT budget over a million dollars. Companies with IT departments of a few hundreds people, 50-70 of which are in the development team. They have quite a few engineers, a few data analysts, some business analysts, some analytics engineers, some data scientists. The some of the business users are “super users”. They can write SQL, they can create their own report on Tableau and they understand data models. The business users understand their business much better than IT. So rather than going back and forth to the IT dept asking for this and that (which btw always takes too long, like months) they ask IT to give them the data, and they will do the analysis and reporting themselves.

Decentralise

In that situation it makes sense to create data products. It makes sense to decentralise. Decentralise the data modelling. Decentralise the reporting function. The analysis function (both business analysis and data analysis). IT supplies the data. The data is delivered via direct access to database. Or through an API, via a web UI. So users go to the Intranet and browse the data catalog there to order the data (they specify some parameters, like months and product categories or account types). And click Download. Within seconds the data file arrives in the user’s desktop in CSV format. They can double click it to open it in Excel.

Data products are also about reports. Again users go to Intranet and open the reports they need. Set the dropdown lists to the right values (like months or product or account) and click Run. Within seconds the report opens showing all the numbers they need. On that report there is an Export button. Users can click it to export all the data on the screen into Excel.

Data products are also about data model. Super users can connect to a database. They can see the views there. They can join the views and query them using SQL. They can ODBC from Power BI or Looker, specify the SQL and get the data out straight into their reports. They ODBC from Excel and Access, and get the data into their financial models. Or their sales models. Or inventory models. Whatever business scenarios they are working on, they can get the data straight from this database.

Some companies call this database a Data Hub. Some call it Snowflake (because it’s on Snowflake). Some call it a Data Warehouse.

In some companies this database is a Kimball star schema. In other companies this database consists of relational tables. They call it a Third Normal Form (3NF) but some of the tables are in the first or second normal form. But it is quite simple to join the tables. Users know that they need to look for column names with the word “key” or “ID” in them. Usually the first column. Those are the joining columns.

Centralise the data engineering

In those companies, although they decentralise the analysis function, the reporting function and the data modelling function, they still centralise the data engineering function. Bringing the data from outside the companies, jumping through the hoops of SFTP and firewall, scheduling data refresh and running automated data quality checks, those are all centrally done within IT. Not in sales, not in finance, not in investment desks, but in IT. IT has something called DataDog which monitors the data platform in both AWS and Azure. IT has Fivetran (which they brought in to replace MOVEit) which do all the data movements from outside and inside the company. IT has Control-M which monitors every single data job in the company. And there are hundreds of jobs running everyday. Whenever a job failed, IT gets an alert, and they go and fix it.

Those tasks are centralised in IT. Business users don’t want to deal with them. They want to focus on the business analysis, data analysis and reporting. They are not equipped to do it like IT. IT is the right place. IT has data engineers. IT has security engineers. IT has software engineers. And they have Testers too. Business users don’t have all that.

Data Mesh

Is that Data Mesh? Strictly speaking no. In Data Mesh, data products are developed by individual departments. Or divisions. Well if you are a corporate giant where every division has their own 200-people IT department, yes by all means decentralise the data engineering. Imagine that you are Siemens. You have aerospace business, electronics business, mining industry, life sciences, transportation, water industry and you also make wind farms. It make sense to decentralise the data engineering. Every division creates their own data warehouse. Every division does their own data ingestions, data modelling and data monitoring. Because you have like many divisions! Each division operates in multiple countries. You have the IT resources (and IT budget!) on every divisions.

You can afford to do the real Data Mesh. Data products are developed by each IT team (because you have IT teams in different divisions in different countries). The data ownership is distributed. Every division do their own data ingestions, data monitoring, data quality checks. Yep, DataDog, FiveTran and Control-M. The whole thing. You have IT team in Singapore. You have IT team in the US. You have IT team in London. You can afford to distribute the data ownership to different business teams and IT teams in different countries. The aerospace business are very different to the life sciences business. The electronics business is very different to the mining business. It makes sense to decentralise.

The role of the central IT team is to do data provisioning. In a data lake. They provide all the data that every division needs in AWS Data Lake. They call it S3. And in Azure Data Lake too. They call it ADLS. And in GCP too. They call it GCS. Yep they are corporate giants with IT budget over a hundred million dollars, perhaps even into the billions. Here’s a guide: IT budget is 2% to 3% of the annual revenue. You can look up the annual revenue of every corporate giant in their annual report. The annual revenue is in billions. Like 50 billion or 100 billion dollars. So 2% of that is 1 to 2 billion dollars. Compare that to your own IT budget. So you understand now that they play a different ball game to most companies. They can afford to do Data Mesh. Most companies don’t have that luxury.

Let’s recap.

Now that we have gone through the reality on the ground, let’s end it with a bit of theory. Data Mesh is about: (thanks to Zhamak Dehghani)

1.????? Data as a product

2.????? Distributed ownership

3.????? Domain-oriented

4.????? Self-service

5.????? Federated governance

Data products are developed by teams who best understand the data. And it is domain-oriented. The electronics business has their own data team (and business teams) who develop their own data products. The life sciences business has their own data team (and business teams) who develop their own data products.

The central IT team provides the data in a data lake. All divisions and all business functions have access to that data lake. Sometimes there are multiple data lakes. In multiple cloud platforms too: AWS, Azure, GCP.

And everything is self service. Each business function shared their data products that other business functions can access in a self-service way.

The data governance is federated. Some data governance are done centrally, some data governance are decentralised.

But if your IT budget is less than half a million dollars and your development team is only 10-20 people, stick to data warehousing. One warehouse. One IT team. IT build the warehouse, business users use it.

If your IT budget is over a million dollars and your development team is 50-70 people, decentralise the analysis function, the reporting function and the data modelling function, but still centralise the data engineering function.

If your IT budget is hundreds of millions dollars and you have multiple divisions/businesses in multiple countries, decentralise everything, and adopt the Data Mesh in full. With federated data governance.

Of course in reality it's not a straightforward cut into 3 distinct models like that. The boundaries are not clear cut. You might have to combine a few things and improvise, creating your own pathways in Data Mesh journey, to suit on your unique circumstances.

As always I welcome your comments and opinions. And corrections. Thank you.