New Data Platforms: The Announced End of ETLs?

Quite provocating, I know... but let's raise the debate

"Paradigm shift in the data world!" That's what one might say when observing new cloud data platforms like Snowflake, BigQuery, Databricks, and their peers. These giants are not only redefining the way we work with data but also raise a crucial question: are we witnessing the end of traditional ETL (Extract, Transform, Load) processes? In the following lines, let's dive into this slightly provocative reflection, the pertinence of which I'll let you judge!

The Impact of (Almost) Limitless Processing Capabilities and (Almost) Marginal Storage Cost

Let's face the facts: cloud platforms have changed the game. With breathtaking processing capabilities and data storage costs that defy any competition, one really wonders if the good old ETL model still has its place. These platforms, like Snowflake and BigQuery, handle data on a scale and speed unimaginable just a few years ago. And all this, while keeping costs under control. So, is this the end of ETLs? Let's look into this question.

Limits of Traditional ETLs

Let's not kid ourselves, ETLs are a bit like those old habits we struggle to let go of. But let's be honest: they require specific, if not advanced, skills, create a software heritage that must be maintained and, in the long run, a technical debt that is not always easy to manage. Not to mention that these processes can often seem heavy and rigid compared to the agile solutions offered by the cloud. Moreover, their existence comes from a world where storage and processing were (very) expensive (like traditional Data Warehouse solutions), and it was preferred to process the data before ingesting it.

Data Cloud Platforms and Automated Ingestion

This is where the new Data platforms come into play, with their range of automated ingestion methods. They offer a seductive alternative, simplifying data integration and making traditional ETL processes almost obsolete. These solutions are both more flexible and more in tune with the current needs of companies, including streaming and real-time.

Evolution of Talent and Cloud Native Approaches

The job market in the data field is also evolving. Today's talents are increasingly turning away from traditional solutions to embrace cloud-native approaches and CI/CD practices. This trend marks a significant turning point, also reducing the appeal and effectiveness of classic ETL processes.

Synchronisation and Data Replication

Ultimately, what we all want is to have a faithful and up-to-date copy of operational data on our Data platforms. And here, cloud-oriented solutions seem much more suitable for this synchronisation or replication of data. ETLs, with their often heavier and less flexible approach, are inexorably losing ground.

Conclusion

So, are ETLs relics of a bygone era? Perhaps not entirely yet, as they are often connected to source systems, which is very practical for feeding a Data platform (if we exclude real-time needs), but it is clear that modern data platforms are redefining the rules of the game.

The transition to more suitable, agile, and cloud-era-aligned solutions is underway.

And what do you think? Are we witnessing the end of ETLs, or is it just one more evolution in the vast world of data?