How do you compare data pipeline designs?

The idea that batch processing is simpler and cheaper has aged pretty poorly. Capturing data live is now generally cheaper than regularly scaning a source (it's called CDC for Change Data Capture if you want to look it up) - and simpler since it's asynchronous nature frees you from the headache of orchestration operations. So, really, focus on your requirements. The choice essentially boils down to producer capabilities and consumer needs: 1) are your input continuous (microservices, event bus, transactional database) or not (batch jobs, third party with periodic availability), and 2) is your consumer periodic in nature (delta-based reporting) or operational/product-driven (user-facing features, ML, alerting, pricing/optimization...).

Your decision between ELT and ETL will define your data storage, analysis, and processing. ETL is generally slower than the other because data is transformed in a server aside while ELT is faster at ingestion due to that loading and transformation can be done simultaneously.

Ultimately, the best choice of architecture will depend on the specific requirements of the application. Here are some additional considerations that you may want to take into account when choosing between Lambda and Kappa architecture: The volume of data: If you are dealing with large volumes of data, Lambda architecture may be a better choice, as it can provide more flexibility in how the data is processed. The cost: Lambda architecture can be more expensive to implement and maintain than Kappa architecture, as it requires two separate processing layers. The skill level of the team: If your team is not familiar with stream processing, Kappa architecture may be a better choice, as it is simpler to implement.

A third option to consider is the “Data Lake House”. It’s becoming increasingly more popular, with Databricks being a nice example. A data lake house provides the necessary infrastructure to harness the advantages of data lakes, enabling scalability, flexibility, and adaptability while addressing the challenges of governance, security, and management in a manner similar to a data warehouse. It aims to get the best of both worlds and combine the advantages of the data lake and the data warehouse which is why it’s rapidly gaining popularity.

it is also important to consider the specific requirements of the application or business case for the data pipeline. The best data pipeline design will depend on the specific needs of the application or business case. Here are some additional insights that I would like to add: The choice of data pipeline design should be made in consultation with stakeholders from all relevant teams. It is important to document the data pipeline design so that it can be easily understood and maintained by others. The data pipeline should be monitored and regularly reviewed to ensure that it is performing as expected. The data pipeline should be updated as needed to accommodate changes in the data sources, destinations, or requirements.