A Data Platform is not just for analytics
Andrew Jones
?? Created data contracts and wrote the book on it. Write weekly to hundreds of smart data folk. Principal Engineer. Father of 2. Brewer of beer. Aphantasic.
Data is at the heart of every meaningful service, and it’s the effective use of that data which builds a product, and a business.
It’s the data from one service that enables another, whether that’s a closely associated microservice, an anomaly detection pipeline, or any other data-driven service.
A data platform has all the components we need to make effective use of this data, but so often we tie them to a specific use case or deployment, leaving the rest of the organisation to reimplement or go without.
By data platform I’m not talking about a data warehouse, or a data lake, or some other large and potentially foreign system. It could be an effective event streaming platform with excellent tooling around it, or a change data capture-like service that can help users implement an outbox, or highly reliable and resilient archiving and backups of data.
Data (Platform) Engineers are great at building these kind of things, but we keep them to ourselves. There’s no technical reason why we can’t implement a service that archives an event stream to a database that works not just with the data we’re responsible for ingesting, but any data that conforms to a known schema or contract, for example.
By opening the data platform up we can enable many more use cases across the organisation, increasing our impact and generating a greater return on investment for all the great tooling and services we are building.
Originally published at https://andrew-jones.com/blog/a-data-platform-is-not-just-for-analytics/.
Cover image from Unsplash.