Why AWS is investing in a zero-ETL future

Data is at the center of every application, process, and business decision. When data is used to improve customer experiences and drive innovation, it can lead to business growth. According to Forrester , advanced insights-driven businesses are 8.5 times more likely than beginners to report at least 20% revenue growth. However, to realize this growth, managing and preparing the data for analysis has to get easier.

That’s why AWS is investing in a zero-ETL future so that builders can focus more on creating value from data, instead of preparing data for analysis.

Challenges with ETL

What is ETL? Extract, Transform, Load is the process data engineers use to combine data from different sources. ETL can be challenging, time-consuming, and costly.

• It requires data engineers to create custom code.
• DevOps engineers have to deploy and manage the infrastructure to make sure the pipelines scale with the workload. In case the data sources change, data engineers have to manually make changes in their code and deploy it again.
• While this is happening data analysts can’t run interactive analysis or build dashboards, data scientists can’t build machine learning (ML) models or run predictions, and end-users can’t make data-driven decisions.

The time required to build or change pipelines makes the data unfit for near-real-time use cases such as detecting fraudulent transactions, placing online ads, and real-time supply chain analysis. In these scenarios, the opportunity to improve customer experiences, address new business opportunities, or lower business risks can simply be lost.

AWS is bringing its zero-ETL vision to life

Zero-ETL makes data available to data engineers at the point of use through direct integrations between services and direct querying across a variety of data stores. This frees the data engineers to focus on creating value from the data, instead of spending time and resources building pipelines.

We have been making steady progress towards bringing our zero-ETL vision to life so organizations can quickly and easily connect to and act on their data. Here are just two examples:

• With?Amazon Redshift ?Streaming Ingestion, organizations can configure Amazon Redshift to directly ingest high-throughput streaming data from?Amazon Managed Streaming for Apache Kafka ?(Amazon MSK) or?Amazon Kinesis Data Streams ?and make it available for near-real-time analytics in just a few seconds.?Customers can connect to multiple data streams and pull data directly into Amazon Redshift without staging it in Amazon Simple Storage Service (Amazon S3).
• Using federated query in Amazon Redshift and Amazon Athena , organizations can run queries across data stored in their operational databases, data warehouses, and data lakes so that they can create insights from across multiple data sources with no data movement. Data analysts and data engineers can use familiar SQL commands to join data across several data sources for quick analysis, and store the results in Amazon S3 for subsequent use.

And just last week we announced the public preview of Aurora zero-ETL integration with Amazon Redshift, to enable near real-time analytics and machine learning (ML) using Amazon Redshift on petabytes of transactional data from Aurora. With this launch, customers can ingest data in hundreds of thousands of transactions every minute in Aurora and can still analyze them real-time in Redshift without having to build labor intensive and expensive ETL pipelines. To learn more read my full blog .

When organizations can quickly and seamlessly integrate data that is stored and analyzed in different tools and systems, they can make data-driven predictions with more confidence, improve customer experiences, and promote data-driven insights across the business.