Data As a Product

In the age of big data, organisations are continuously looking for new ways to manage and leverage their data assets. Data Mesh has evolved as a game-changing paradigm change in data management, enabling scalable data democratisation and driving organisational agility through a decentralised and domain-oriented approach. This essay delves into the Data Mesh idea and its transformational potential for revolutionising data management practises.?

Data Mesh and Starburst:??

In a data mesh architecture, the focus is on decentralizing data ownership and enabling self-service access to data within an organization. Here's how Starburst Presto can fit into such an architecture:?

1. Data Domain Ownership: In a data mesh, each data domain or team takes ownership of their data products. They are responsible for defining and managing their data infrastructure, including data storage, processing, and access. Starburst Presto can act as a central querying layer that enables teams to query and access their data products efficiently.?
2. Data Storage: Data mesh often involves storing data in distributed data storage systems like data lakes or cloud-based storage solutions. Starburst Presto can connect to various data sources, including data lakes, data warehouses, and other data platforms. It allows teams to query and join data from different sources using standard SQL queries.?
3. Data Access and Querying: Starburst Presto provides a distributed SQL query engine that allows teams to query data across multiple data sources. It supports federated querying, which means you can define connectors to different data systems and query them as if they were part of a single system. This enables teams to access and combine data from different domains seamlessly.?
4. Scalability and Performance: Starburst Presto is designed to handle large-scale data processing and analytics. It offers distributed query execution, parallelism, and advanced query optimization techniques, which make it suitable for handling the scalability requirements of a data mesh architecture.?

By incorporating Starburst Presto into a data mesh architecture, organizations can provide teams with self-service access to their data while ensuring data ownership and decentralized control. Presto's distributed query capabilities and scalability contribute to the efficient querying and analysis of data from various sources within the data mesh environment.?

Data Products in the Data Mesh?

Data Mesh aims to clarify and prescribe that the ownership and architecture of data products belongs to the domain, but further that data is treated as a first-class product across the organization. This is a drastic mental shift, wherein data is no longer treated as a by-product of activities that the business engages in, but as a business product in its own right. It’s been shown time and again that there is inherent product-level and game changing value in data; data is a key value-driver that should aggressively direct business decisions. Businesses should therefore invest in creating and managing that data with the same care and forethought that they do other products and services.?

What’s this DATSIS??

Data Mesh’s goal is to allow end users easier access to data so that they can derive business value faster and more reliably. Moreover, Data Mesh clarifies the roles that the domain and the central IT team play, which helps avoid any “shadow IT” either in the domains or among the analytics folks. To that end, the ideal data product has several qualities that drive this goal as well as overall data governance. The goal is to make data:

• Discoverable: End users and other domains need to be able to discover and access a given data product?
• Addressable: The data should have a straightforward and documented way of being programmatically accessed, e.g., via SQL?
• Trustworthy: End users should be able to understand the level of data quality and ideally view the provenance (lineage) of the data so they can be confident in any analyses using the data product?
• Self-describing: Any end user outside the domain which produces the data product should have all of the information they require to use the data?
• Interoperable: Governance should ensure that the data complies to any inter- or intra-domain standards or regulations, so the end user can confidently use the data without concern?
• Secure: Data products should fold any authorization into the access control provided by the data mesh experience plane, which is where data product consumption occurs?

The handy acronym DATSIS allows us to remember the key elements of a data product, and the domains producing these data products should design their products to conform to these standards.?

Reference link?

https://www.starburst.io/blog/data-mesh-and-starburst-data-as-a-product/?