How does Zero ETL differ from a  traditional ETL approach?
Credit: DALLE Image Generation from OpenAI

How does Zero ETL differ from a traditional ETL approach?

Recently, AWS released Zero ETL at Re: Invent 2022, with Microsoft, Google, and data warehouse providers Snowflake and Data Bricks following suit.

What is it? A set of integrations that eliminates or minimizes the need to build ETL data pipelines. Zero ETL is a misnomer. A better way of describing what it does is "Zero EL."

How does Zero ETL differ from ETL? ETL, which stands for?extract, transform, and load,?is a data integration process combining data from multiple data sources into a single, consistent data store loaded into a data warehouse?or other target system.

  • Extract?data?from legacy systems.
  • Transform or cleanse the data to improve data quality and establish consistency.
  • Load data into a?target database.

How is this achieved? Zero-ETL has the transactional database do the data cleaning and normalization before automatically loading it into the data warehouse. It’s important to note that the data is still relatively raw.?This tight integration is possible because most zero-ETL architectures require the transactional database and data warehouse to be from the same cloud provider.

Pros: Reduced latency. Reduced or eliminated duplicate data storage. One less source of potential failure.

Cons: There needs to be more ability to customize how the data is treated during the ingestion phase—some vendor lock-in. It doesn't eliminate later transformations that may need to be done to make the data consumable.

I hope you enjoyed reading about this brief overview as much as I did writing it. Next week, I will write about more traditional data pipeline methodologies and their pros and cons, including ETL, ELT, EtLT, and reverse ETL.



Abul F.

Data Engg & MLOps for Risk models @ BMO | Scaling Data teams | ETL, Data Products, Data Platforms, MLOps, GenAI, Project management

10 个月

Great article. Can you share some more detail on how zero-etl enables real-time dataflow from transactions to data warehouse? From your article as mentioned as “EL”, I am guessing it is not as useful as no data modelling transformation is applied.

回复

要查看或添加评论,请登录

Mark Koss的更多文章

  • Why banks need the Banking Industry Architecture Network (BIAN) model

    Why banks need the Banking Industry Architecture Network (BIAN) model

    Historically, banks who wanted to connect to a new fintech service or replace legacy systems faced several roadblocks…

  • What is a semantic layer and why do you need it

    What is a semantic layer and why do you need it

    We have all been there. Your numbers differ from another team's on a critical sales campaign or key metric, and you…

    1 条评论
  • Why Snowflake versus SQL Server?

    Why Snowflake versus SQL Server?

    As an expert in database architecture, I am often asked what the business benefits are of moving from SQL server to…

  • Cascade Bicycle Club - Why I Love Seattle

    Cascade Bicycle Club - Why I Love Seattle

    Are you new to Seattle and looking to meet great people? Do you want to experience the beauty of the Pacific Northwest…

    6 条评论
  • How to create an effective BI dashboard

    How to create an effective BI dashboard

    You have identified your dashboard's audience, clarified the questions it will answer, and identified the relivant…

    3 条评论
  • The lost art of data storytelling

    The lost art of data storytelling

    As dashboard technologies like Power BI and Tableau become more powerful and dashboard design becomes more efficient…

    2 条评论
  • What is a data vault, and when do you use it?

    What is a data vault, and when do you use it?

    Companies transform data through an ETL (Extract, Transform, and Load) into an enterprise data warehouse to analyze and…

    1 条评论
  • Other Variants: ELT, EtLT, and Reverse ETL

    Other Variants: ELT, EtLT, and Reverse ETL

    What is ELT, and how is it different from ETL? ETL transforms the data before loading it into the data warehouse using…

    5 条评论

社区洞察

其他会员也浏览了