From Data Catalogues to Data Marketplaces: Unlocking the Full Potential of Your Organisation's Data with AWS

Introduction

In today's data-driven world, organisations are drowning in information but thirsting for insights. The sheer volume of data generated daily is staggering – according to IDC, the amount of data created over the next three years will be more than the data created over the past 30 years. Yet many companies struggle to extract value from this data deluge. For a number of years now, many organisations have resorted to traditional data management and cataloguing techniques to govern and control their increasing sprawl of data.?In order to then tap into it with other ancillary tooling, dashboards and advanced analytical capabilities.

Yet, the key to unlocking the huge potential that lies in many organisations' data requires said organisations to move beyond traditional data catalogues and embrace the notion of creating a modern data marketplace for their employees, partners and suppliers. This evolution not only introduces the opportunity for organisations to enhance internal decision-making. But also opens up new avenues for 3rd party collaboration and revenue generation that many organisations are yet to fully leverage as the world seemingly races towards the next AI enabled juggernaut.?

However, becoming data driven, AI powered and everything in between is a journey. It takes investments in your people's skills, the drastic changing of mindsets, new technology capabilities and the creation of new processes that embrace automation wherever possible. This means clearly defined policies and controls that are enablers for innovation. Not meaningless roadblocks for governance and ivory tower naysayers.?

Data catalogues have been part of enterprise technology DNA for numerous years now. However, to get the most from your data, merely having a 2-dimensional interface into your data is no longer enough. This is where the notion of a connected data marketplace comes into play.

A data marketplace transforms the static nature of traditional catalogues into a dynamic, interactive ecosystem. It's not just about listing what data exists; it's about creating a vibrant, self-service environment where data can be discovered, understood, and leveraged with ease. This marketplace approach fosters a culture of data democratisation, where insights are no longer siloed within specific departments but are accessible to all who need them.

Moreover, a well-implemented data marketplace can serve as a springboard for innovation. It enables cross-functional teams to collaborate more effectively, sparking new ideas and approaches. It also opens up possibilities for external collaboration, allowing organisations to tap into broader ecosystems and potentially uncover new revenue streams.

As we delve deeper into the concept of data marketplaces, it's crucial to understand the journey an organisation must undertake to fully realise this vision. This journey is not just about implementing new technology; it's about fostering a data-driven culture, developing new skills, and reimagining processes.

In the following sections, we'll explore the key stages of this journey, from establishing strong internal data governance to facilitating secure external collaborations, and ultimately, to potentially monetising your data assets. We'll discuss how each stage builds upon the last, creating a robust foundation for data-driven decision making and innovation.

So What Exactly is a Modern Data Marketplace?

Before we delve into the journey of data maturity, it's crucial to understand what we mean by a 'modern data marketplace'.?

A modern data marketplace is a dynamic, secure, and governed environment where data is treated as a valuable asset that can be easily discovered, accessed, and shared both within and outside an organisation. Unlike traditional data catalogues, which often serve as static inventories of data assets, a data marketplace facilitates active interaction with and exchange of data.

Key characteristics of a modern data marketplace include:

1. Self-service discovery: Users can easily search, browse, and understand available data assets without needing to involve IT departments for every request.

2. Automated governance: Built-in controls ensure that data access adheres to regulatory requirements and organisational policies, reducing the risk of data misuse or breaches.

3. Data quality assurance: Mechanisms are in place to ensure the reliability, accuracy, and timeliness of data, often including user ratings and reviews.

4. Seamless integration: The marketplace integrates smoothly with various data storage, processing, and analytics tools, allowing users to work with the data directly in their preferred environments.

5. Collaboration features: Users can share insights, provide context, and collaborate around data assets, enhancing the overall value of the data.

6. Monetisation capabilities: For more advanced implementations, the marketplace may include features to package, price, and sell data products to internal or external consumers.

7. Usage analytics: The platform provides insights into how data is being used, helping to identify valuable data assets and opportunities for improvement.

By implementing a modern data marketplace, organisations can transform data from a static resource into a dynamic, value-generating asset. This approach not only enhances internal decision-making but also opens up new possibilities for data-driven innovation and revenue generation.

However, in order to achieve this your organisation's needs to make decisions about having the right enablers in place to underpin your journey across the data maturity curve.?

Getting the Right Enablers in Place

So how do organisations make the right journey, at the right speed that matches their risk appetite and investment desire? And who do they make the journey with in terms of technology providers??

Personally, I’ve always had a soft spot for Amazon Web Services. Partner friendly, great technology solutions and capable of delivering cloud based services that can host critical national infrastructure. I also think that their data offerings are becoming increasingly well rounded and complementary of their generative AI services provided by solutions like Bedrock and Bedrock Studio. As such, over the course of this blog, I will cover 3 stages of maturity and discuss 3 core services that AWS provides organisations seeking to govern, share, trade and monetise their data. Namely;

Step 1: AWS Datzone - Level 1 Foundations, making sure your data is governed, owned and accessible in a trustworthy manner.?

Step 2: AWS Clean Rooms - Level 2 Building Blocks enables your business to begin to explore how it can trade, barter and enrich your data with insights provided by 3rd party partners and suppliers. Meaning you can get away from sending troves of data via email and the deathly deluge of spreadsheets.?

Step 3: AWS Data Exchange - Level 3 Panacea where you can be assured that your data is of the highest quality. To such an extent that you now feel confident in your ability to monetise it, whilst satisfying your internal compliance teams requirements to protect mission critical data-sets, whilst opening up new opportunities to generate revenue for your business.

The Data Maturity Curve: A Journey to Data-Driven Decision Making

The path to becoming a truly data-driven organisation can be visualised as a journey along a data maturity curve. This curve represents the progression from basic data management to advanced data monetisation. As organisations climb this curve, they unlock new capabilities and sources of value from their data assets.

The journey consists of three primary stages:

1. Internal Data Discovery and Governance

2. Secure Data Collaboration with External Partners

3. Data Commercialisation and Monetisation

Let's explore each stage in detail, examining how modern tools and platforms are enabling this transformation.

Stage 1: Internal Data Discovery and Governance

The first step in the data maturity journey involves breaking down internal data silos and establishing robust governance practices. Many organisations struggle with fragmented data landscapes, where valuable information is scattered across various departments and systems, making it challenging to gain a holistic view of the business. This is where concepts like Data Mesh and Data Fabric can come into play and this is something that I wrote about several weeks back .?

On this occasion though…enter AWS DataZone , a service designed to tackle this very challenge. DataZone acts as a centralised hub for data discovery, access, and sharing across business functions and domains within an organisation. It provides a user-friendly interface that allows employees to search for and access relevant data, regardless of where it's stored in the organisation.

Key features of AWS DataZone include:

- Comprehensive data cataloguing

- Fine-grained access controls

- Data lineage tracking

- Integration with existing data storage and analytics tools

By implementing DataZone, organisations can:

1. Improve data discoverability, enabling employees to find the data they need quickly

2. Enhance cross-functional collaboration by breaking down data silos

3. Maintain compliance with data regulations through built-in governance controls

4. Ensure data quality and reliability through standardised metadata management

For example, Natera, a global leader in cell-free DNA (cfDNA) testing, dedicated to oncology, women’s health, and organ health have adopted AWS DataZone and applied it as part of a data mesh architecture. In doing so, they were able to shift from an overly saturated in demand central data team who were struggling to keep up with access requests, towards a self-service data operating model where they were able to measure the quality and decipher the origin of their data immediately.?

Invariably, as organisations master this stage, they lay the foundation for more advanced data initiatives. Clean, well-governed data becomes the backbone for improved decision-making and enhanced customer experiences. Making trusted data available to your internal stakeholders is an absolute necessity for every business. However, as data becomes increasingly verbose, varied and voluminous the opportunities to round it off with additional insights from trusted 3rd parties can be an absolute game changer for many organisations.?

Let’s progress to stage 2.?

Stage 2: Secure Data Collaboration with External Partners

As organisations become more adept at managing their internal data, many realise that combining their data with external sources can yield even more powerful insights. However, data sharing with external parties raises significant privacy and security concerns.

AWS Clean Rooms addresses these challenges by providing a secure environment for data collaboration. This service allows organisations to join their data with that of partners or third parties without exposing raw data to either party.

Key features of AWS Clean Rooms include:

- Secure, permission-based data access.

- Customisable data clean room creation.

- Built-in privacy-enhancing technologies.

- Seamless integration with AWS analytics services.

The benefits of using AWS Clean Rooms for data collaboration include:

1. Enhancing data insights by combining datasets securely.

2. Maintaining control over sensitive information.

3. Enabling new partnership opportunities.

4. Improving the accuracy of analytics and machine learning models.

Global TV Network operator Fox uses AWS Clean Rooms in order to support their advertising clients in figuring out how best to leverage more data sources to optimise their media spend across their combined portfolio of entertainment, sports, and news brands, which reach ~200M monthly viewers. Fox are reportedly using AWS Clean Rooms to enable easy and secure data collaborations on the AWS cloud that will help their advertising clients unlock new insights across every Fox brand and screen while protecting consumer data.

This stage represents a significant leap in data maturity, as organisations begin to realise the value of data as a collaborative asset rather than a proprietary secret. By the very nature of these types of agreements organisations need to be increasingly open to new partnerships that they may have previously dismissed. Invariably, Clean Rooms can also be extremely useful for sharing highly sensitive data that may be traded for investigations, merger & acquisitions or legal processes on a global stage. All whilst data sovereignty requirements are abided by.?

So we’ve governed our data. Made it accessible internally and now opened up the possibility of sharing data externally. Now let's make some real $$$ out of your strategic data assets.?

Stage 3: Data Commercialisation and Monetisation

The final stage of the data maturity curve involves recognising data as a product in its own right – one that can be packaged, marketed, and sold to generate new revenue streams.

AWS Data Exchange facilitates this process by providing a secure marketplace where organisations can list, sell, and purchase data products. Companies like Experian and Morgan Stanley have already leveraged this platform to monetise their data assets, with many other global brands following suit.?

Key features of AWS Data Exchange include:

- Secure data packaging and delivery.

- Flexible licensing options.

- Integrated billing and payment processing.

- Access to a wide range of potential customers.

The benefits of data commercialisation through AWS Data Exchange include:

1. Creating new revenue streams from existing data assets.

2. Reaching a broader market of potential data consumers.

3. Standardising data products for easier consumption.

4. Accelerating time-to-market for data-driven solutions.

For instance, organisations such as Morgan Stanely, Equifax, Experian and Progonos Health are already selling data products on AWS Data Exchange with some of these solutions starting with annual subscription fees of ~$13,000-$29,000 P/A. Sell 20 of those per year and you will pretty much have funded your own central data team and turned them into a profit centre…not a cost centre.?

However, data commercialisation comes with its own set of challenges, including ensuring data quality, managing customer expectations, and navigating complex regulatory landscapes. Organisations must carefully consider these factors when venturing into data monetisation.

Progressing Through the Data Maturity Curve

It's important to note that progression through these stages is not always linear. Organisations may find themselves at different stages for different parts of their business. The key is to view this as a continuous journey of improvement and value creation.

Each stage builds upon the capabilities developed in the previous one:

- Strong internal data governance (Stage 1) is crucial for effective external collaboration (Stage 2)

- Secure collaboration experiences (Stage 2) can inform and refine data product offerings (Stage 3)

- Insights gained from data monetisation (Stage 3) can feed back into improving internal data practices (Stage 1)

Organisations should approach this journey strategically, assessing their current capabilities and planning incremental improvements. It's not necessary – or often even possible – to jump directly to Stage 3. Instead, focus on mastering each stage before moving to the next. However, if you need to move faster, look at setting up a series of Pathfinding Projects to prove or disprove the possibilities for each stage in your business and ideally look to execute these in different parts of your organisation to ensure that the initiatives don’t trip over one another.?

Conclusion

The evolution from traditional data catalogues to modern data marketplaces represents a fundamental shift in how organisations view and utilise their data assets. By progressing along the data maturity curve – from internal data discovery and governance, through secure external collaboration, to data commercialisation – organisations can unlock the full potential of their data.

This journey not only enhances decision-making and operational efficiency but also opens up new avenues for innovation and revenue generation. In an increasingly data-driven world, organisations that master these capabilities will find themselves with a significant competitive advantage.

As you consider your organisation's position on this curve, ask yourself:

- How effectively are we currently utilising our internal data?

- What opportunities exist for secure data collaboration with partners or customers?

- Could our data assets provide value to other organisations?

The answers to these questions will guide your path forward in the new data economy. Remember, becoming truly data-driven is not a destination, but a continuous journey of learning, adaptation, and value creation.