Data Mesh: How IT is starting to drive organizational design

In the ever-evolving technological landscape, the importance of harnessing and understanding data is paramount (sound familiar?). Centralized data monoliths have been the mainstay, but as organizations scale and diversify, the siloed approach to data becomes a bottleneck (equally not new). Also not new is the concept that IT is a metaphor for a roadblock in the business user’s mind.

Enter the Data Mesh data platform architecture. The data mesh architecture helps business users create and manage data products without depending on IT resources to do it. This novel concept sounds like another ebb and flow of centralized to decentralized and back again. Yet, if we reframe this situation slightly, a new concept emerges. Is data mesh a data platform architecture or an approach that enables IT to help shape the design of the business?

What is a Data Mesh?

In short, think of the data mesh as a city. City planners and engineers build and maintain the infrastructure, and its citizens use it at will. In the case of data, IT builds and maintains the data platform but enables the business to leverage it without depending on IT resources to build value-enabling data products.

So, what does this have to do with organizational design?

Despite our best efforts as technologists, we have not yet constructed a way for non-technical users to build an exhaustive list of data products. Amazon Q with QuickSight is a monumental step forward in enabling analytics dashboarding using natural language and Generative AI, but Q does not yet capture the full range of data analytics and AI needs of business users. Nor do existing no-code/low-code platforms (though these are continuously getting better and better).

?So, where does this leave us? The Data Mesh philosophy creates constructs enabling data product ownership to live within the business. Back to our city planning metaphor, the city is built by IT, and its business citizens become citizen developers imagining and building the data products they need without suffering the delays of heavily contended IT resources to become available (supply and demand ring a bell?). If only the story ended here. It starts to unravel with one question:

What happens when I exceed the skills of a citizen data product developer?

?Today’s data product needs span the gambit from dashboards to building (not using) machine learning models. And somewhere along the way, the business will find that its decentralized autonomy hits a skillset wall, leaving the business to source key roles in data product design and development. Roles like Data Scientists, Machine Learning Engineers, Data Engineers, Developers, and Data Governance Stewards. This is where IT dons a new hat and leans into advising the business on organizational design.

How IT advises the business on something other than technology and security.

For decades, most businesses have looked to the IT function to be both data platform and data product builders. In a decentralized data platform world, where key data products are federated, IT steps a bit to the side. However, those decades of experience have taught IT a thing or two about understanding and building data products. This is where IT gets to lean in as an advisor as business functions spin up their data product manufacturing capabilities.

Role Hydration

Where demand necessitates, it may make sense to fully hydrate certain data product development roles within the business function itself. These roles are filled by individuals who exist within the business unit. IT plays a key role in advising the business on what demand shapes the need for a full-time role and what skillsets the business should seek. IT can also advise on the longer-term data product roadmap and can help shape the growth trajectory of the team and its requisite skill sets.

Shared Resources

In a federated data platform environment, business units may operate their data product construction in isolation. While the data mesh and underlying catalog and semantic layers may make those data products readily discoverable and available throughout the enterprise, that may not mean the data gurus within those business units are talking amongst themselves. Remember the city planner metaphor? City planners often build capabilities to understand how the city is functioning (cameras, traffic flow detection, infrastructure gauges, etc.). This enables them to identify hotspots and trends that inform the planners about the system.

A well-designed data mesh platform uses a similar construct: observability and telemetry. Pair IT’s ability to monitor the overall data platform with their “hub and spoke” support model (business units each partnering with IT), and you have asymmetrical information that enables IT to identify when a shared resource model might just meet the need. Where demand is insufficient for a single business unit to support a full-time role, two or more business units can share the cost and productivity of a data product developer team. IT has seen this setup before – it has existed since day one, as many in IT often serve more than one business unit customer. Here IT can once again advise on the constructs of building shared teams to ensure each customer business unit receives the attention and productivity required.

Wherefore art thou IT?

Let’s face it – there are many reasons why having data product developer roles within the business might not work: resource shortages, budget limitations, a culture of de-duplication, training, running and maintaining challenges, etc. In these scenarios, IT can build dedicated teams that are outsourced to business units. IT can advise on what structure these teams should take and how to manage the data product development lifecycle. With expertise in bill-back constructs and knowledge of the overall data ecosystem, these shared IT teams can operate independently of the data mesh platform team to ensure a “separation of concerns” while remaining managed by the IT function.

Conclusion

The data mesh is not just a technological shift—it's a fundamental rethinking of how organizations treat data. For technology executives, it's essential to recognize that this paradigm shift in data architecture has equally profound implications for organizational design. By understanding and embracing these dual facets, businesses can harness the power of their data more effectively, innovatively, and responsively.

Note: Data mesh is still a growing concept, and its implementation can vary based on organizational needs. It's always essential to collaborate closely with experts to tailor the approach to your specific requirements. Please contact me if you’d like to discuss how your organization can successfully implement data mesh platforms both technically and organizationally.