Building User-Centric Data Products

[TL/DR: A super article that lets you know data products are more about the right value-driven company culture and less about technology]

There is increasing interest in data products within many CDO/CIO functions globally. These are often associated with data mesh, a glowing hot methodology that the hype suggests (not without merit) to be a solution to many failing data programmes.

But given their popularity it is surprising that few know what they are (let alone built any), and even amongst data practitioners there is debate about the fundamentals of what they actually are.

I have been fortunate to be involved in several clients (both large and small) that have had to wrestle with the practicalities of implementing these, and this paper (and later ones covering data mesh) is my attempt to share some of my experiences from these assignments.

My view is that whilst data product are a viable organizational pattern, there are many over-expectations around them simply because the approach is relatively new, as well as misunderstanding that the challenges are mainly technology (spoiler: company cultural is much more significant). Hopefully in reading this guide you can be better prepared for them in your own data journey.

Why do this?

Many organizations have a wealth of valuable data, which whilst useful to themselves, by innovating could develop into a useful revenue stream by finding new consumers of it. In this paper, I will cover a background to data products and their creation process.

Data product takes a tangent to the received wisdom that centralising your data in one monolithic data store is a good idea. In reality these projects often decrease the quality and usefulness of data to those who need it.

What are data products?

Data products are simply packages of data together with associated services, applications and metadata required to make them useful. These can include services or applications for data entry, validation, search and aggregation.

Whist some practitioners describe data products having no service element, I disagree with this assertion and suggest these be treated no different to standard software products.

In the same way a traditional software application is not just application code, it is code and data. Data products typically also have at least a basic bit of logic wrapped into them (even if it is just metadata). As such they need to be planned, developed, tested, marketed and distributed.

Where data products diverge slightly from software products is around the consumer’s perception that the value is derived mainly from the data itself, rather than the services being performed on the data (such as through an app). This can be as much influenced by marketing than reflect reality, so in designing these to think of them only in a data lens (rather than the services around them) is in my opinion a mistake.

Types of data products

Whilst services tend to be finite, data is unbounded and has the potential to be infinite. This is both a strength and weakness of data products, making them hard to design. Data products therefore have unique characteristics and can be categorised, based on increasing margin, sophistication and also risk.

• Zero Order Products (displacement): Here the consumer is replacing data intensive activity they could do themselves with the product, and the seller of the product is taking on that activity in a standardised manner. Often the value is around the avoidance of doing an activity the buyer wishes to avoid (or stop performing themselves). Examples of this can range from supplying a specific source of raw data to full data hosting providers.
• First Order Product (linear): Here the product is performing some analytical, validation or data management capabilities on the data but is not otherwise integrating or enriching in a substantial way. Examples can range from providing a graph for a specific purpose to full regulatory reporting application.
• Second Order Products (derivation): Here the product is creating something new by combining data sets (from consumer’s own data, product suppliers’ internal data and/or external data) in a new innovative way and provides an enhanced level of insight on the combined data set. Examples range from credit rating application to data enriched with LLMs

How are data products monetized?

Like any product, a consumer is interesting in purchasing something because it is useful (or perhaps cynically), perceived useful to them.

It is important then to consider the data-related motivations for doing this. These could include the following:

• The consumer does not want to do this data activity themselves anymore.
• The product can do it cheaper and/or more reliably than the consumer themselves.
• The consumer does not yet have skills or capabilities to do themselves, or it is an area they don't yet want to focus on (but it might be something they do for themselves later)
• The consumer cannot get the insight on the data themselves, either because the skills in doing this is unobtainable to them and/or the data is unobtainable to them

The more complicated the data product, the higher the perceived value as they often require specialist roles such as data scientists to perform. Generally, whilst these have a better margin on sales, they also have higher intrinsic costs and risks of failure. Particularly if the company developing them has little skills in the areas.

For this reason, it would be a mistake to think only think that complex products are worth designing: If a company has mature data-intensive business processes in one area these are great candidates to build a product around both rapidly cheaply and safely

Conversely, it is only worth taking the risk of the disruptive derived data service products, if the developer takes data management and governance extremely seriously, and has experience in innovating and delivering in an agile way as these are likely subject to rapid and regular change to be commercially viable.

How are data products created?

Data is often poorly understood in organizations, which makes finding products around them even harder to innovate. It is therefore essential to have at least a basic level of data governance in place, including a business glossary and access to data stewards.

Even with this foundation the process of innovation in data context is not easy. Some groundwork is required.

• Data products are equally about processes: Data primarily has meaning within the context of a business process so looking at these is a good place to start. Brainstorm what outcomes your core processes and associated data domains could be useful to build into a product, and then look at the critical data elements (CDE) and services, as well as opportunities to automate and improve quality.
• Data products are about needs: Generating ideas around a specific technology outcome is never a good idea: The goal of products is not to be, say on Snowflake, or to perform generative AI, but to provide some benefit. Focus rather on the things which drive business needs such as regulatory challenges, industry trends, pain points, competitors etc.
• Data products are about accepting a level of risk: When it comes to disruptive products, advanced analytics, generative AI and Machine Learning may appear appealing and profitable. However with complexity comes risk and potentially long development lead times. Sometimes the best soltuons are those that are simple and quick to market.

What is the process of innovation?

The process of innovation is best done with a cross functional group and requires the group to put themselves in the mind of a user or customer. At this point have a broad divergent group as we want to generate as many ideas as possible. Data analysts may not be best suited to lead this, rather a UX designer may be better as the questions asked can be very similar to that in building service offerings such as:

• Are the unexpected or new experiences from same data?
• What is right outcome for user?
• What data offerings can be tailored to a potential customer need?
• What is usable (and profitable)

A dynamic “push” often helps innovation, so rather than planning for a say a longer 6-month process, it is best done via several targeted short (2-ish) week accelerators, to produce a Minimum Viable Product (MVP) focused around a real and specific business outcome. Each accelerator should have at least 2 weeks lead up to mobilise the team (and ensure all data available), as well as 2 weeks wind-down to present findings, learnings and recommendations.

It is essential all this is done collaboratively, and you have all the people (business, technical and importantly data architects from all the data sourcing areas) engaged during each accelerator. These accelerator teams can work alongside more strategic goals and feed into them, but need a level of independence, and must also be decoupled from any dependency to them.

The idea then is to converge quickly on those which have a low time to market (irrespective of risk or complexity). Here a data analyst can take a lead as the questions are more data centric:

• What is currently missing?
• Do we have it?
• Can we acquire it?
• Can we develop this quickly?

Once specification is complete, an agile process can be established using data centric technologies such as Snowflake or GCP to rapidly develop, test, iterate and deploy a marketable data product.

From data products to data democratization

As part of the process of creating data products, you will find that you develop a better understanding of your data in a way traditional organization hierarchies fail to provide. What you have done is facilitate the process of data democratization and data enablement!

Data democratization is empowering those who are most impacted by an organization's data in all aspects of its governance, including how it is defined managed and used.

Those involved in the day-to-day of data (including these working on the front end directly with customers) can have good practical knowledge on data, its limitations, workarounds and shortcuts are usually well placed to know what is useful and can be trusted.

A cross-disciplined group of such SMEs pulled together to build a product which is making money may be a great foundation to keep together longer term. And as you build more and more of them you may want to move away from traditional top-down hierarchies of data domains, functions or geographies to one aligned to a network of products.

Of course, not everything can be solved by products and there is still a strong need for centralised teams of data professionals for optimising, promoting standardisation (where it is most needed) and managing common data (where it is most needed), but by taking a more product aligned view you can help answer the question “how valuable is my data”.

This organisation realignment is also the basis of “data mesh” which I will explore further in later articles.

If you found this article interesting, please do leave a comment and/or like! I am passionate about data, so if you have any data related challenges in your company (large or small) and would like my help (or even better, hire me!) please contact me on+44 7710 435227 / [email protected]