Predict & Influence Muses Series 001: The Chase of the Next Data Platform/Technologies
Throughout our engagements with customers and other large organisations, we often learn that almost all organisations are chasing for the "Next Big Things" in data platforms/technologies, thanks to the software & solution principals (we are partly guilty as well :-) and the consultants alike. These are the kind of technologies that we have been asked to advise and implement:
1) data warehouse, data mart - since our inception (2014) till now
2) data lake - since our inception till now
3) data lakehouse - since 2020 till now
4) real-time data streaming - since 2018, especially when dealing with real-time transactions (fraud detection or rapid customer/system actions are required)
领英推荐
5) data mesh - since late 2021?
However, the executives and management may not realise that these new, shiny piece of data platforms or technologies may or may not solve their immediate needs. Under the umbrella of "visionary", "build for the next generation" (which is not wrong but may not be applicable to ever-changing technologies), executives tend to select the latest and greatest (and likely the most expensive), with the wishful thinking of "build, and we should get the benefits x years from now" with a beautiful prediction of ROI.?
The legend was that a major international bank started a "big data revolution", with 50 million USD investment back in the early 2010s, and hired 50 data scientists. The goal then was "bring all the possible data into the data lake, and our smartest data scientists will be able to find values out". This strategy backfired, as after 2 years there is no use case or value able to derive from the data lake. The team was dismantled.?
A better approach, as we observed and experienced, is to build the data platforms (whatever the technologies may be) based on the needs of initial use cases. Along as the data platforms (and the implementers) are able to ensure data health and "trustability" of the data (is this data safe, is this data of high quality, is this data governed?) the users of the data platform will have high confidence in using it, and therefore the organisation as a whole will be benefited from the staged investment.?