登录查看更多内容

Design data pipelines for streaming and real-time data sources?

由人工智能和领英社区提供技术支持

Streaming and real-time data sources, such as sensors, web logs, social media, or IoT devices, generate large volumes of data that need to be processed and analyzed continuously. Data engineering is the skill of designing and building data pipelines that can handle these data sources efficiently and reliably. In this article, you will learn some common data pipeline design patterns for streaming and real-time data sources, and how to implement them using popular tools and frameworks.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

Rapha?l MANSUY

Data Engineering | DataScience | AI & Innovation | Author | Follow me for deep dives on AI & data-engineering
Matt M.

Revenue Operations Leader | GTM Engineer | Revenue System Architect

1 Lambda architecture

One of the most widely used data pipeline design patterns for streaming and real-time data sources is the lambda architecture. The lambda architecture consists of three layers: the batch layer, the speed layer, and the serving layer. The batch layer processes historical data in batches using tools such as Hadoop, Spark, or Hive. The speed layer processes incoming data in real time using tools such as Kafka, Storm, or Flink. The serving layer combines the results from both layers and provides a unified view of the data using tools such as Cassandra, HBase, or Elasticsearch.

添加您的观点

Rapha?l MANSUY

Data Engineering | DataScience | AI & Innovation | Author | Follow me for deep dives on AI & data-engineering
举报内容
When designing data pipelines for streaming and real-time data sources with a focus on Lambda architecture, you can leverage Apache Kafka for real-time data processing, Kafka Connect for data ingestion and propagation, Debezium for low-latency CDC streaming, and CDC techniques for capturing and streaming data changes. Change Data Capture (CDC) is a technique for tracking and capturing changes in data sources, such as databases, and streaming those changes to other systems for further processing or analysis CDC (Change Data Capture) can be used in Lambda architecture to accelerate and streamline data consolidation from various sources into a Kafka message queue, which then feeds the batch and speed layers of the architecture.

已翻译

赞

加载更多内容

2 Kappa architecture

Another data pipeline design pattern for streaming and real-time data sources is the kappa architecture. The kappa architecture simplifies the lambda architecture by eliminating the batch layer and relying only on the speed layer. The speed layer processes all data, both historical and incoming, in a single stream using tools such as Kafka Streams, Spark Streaming, or Flink. The serving layer provides a view of the data using tools such as Cassandra, HBase, or Elasticsearch. The kappa architecture reduces the complexity and latency of the data pipeline, but requires more robust and scalable streaming tools.

添加您的观点

加载更多内容

3 Micro-batch architecture

A third data pipeline design pattern for streaming and real-time data sources is the micro-batch architecture. The micro-batch architecture is a hybrid between the batch and the speed layers of the lambda architecture. The micro-batch architecture processes data in small batches, typically in the order of seconds or minutes, using tools such as Spark Streaming, Flink, or Beam. The serving layer provides a view of the data using tools such as Cassandra, HBase, or Elasticsearch. The micro-batch architecture balances the trade-offs between latency and throughput, and allows for more complex and stateful processing of the data.

添加您的观点

4 Stream-stream join architecture

A fourth data pipeline design pattern for streaming and real-time data sources is the stream-stream join architecture. The stream-stream join architecture is a variation of the speed layer of the lambda or the kappa architectures. The stream-stream join architecture processes multiple streams of data in parallel and joins them in real time using tools such as Kafka Streams, Spark Streaming, or Flink. The serving layer provides a view of the data using tools such as Cassandra, HBase, or Elasticsearch. The stream-stream join architecture enables more advanced and dynamic analysis of the data, such as correlating events, detecting anomalies, or enriching data with external sources.

添加您的观点

5 Stream-batch join architecture

A fifth data pipeline design pattern for streaming and real-time data sources is the stream-batch join architecture. The stream-batch join architecture is a variation of the lambda architecture. The stream-batch join architecture processes one stream of data in real time using tools such as Kafka, Storm, or Flink, and joins it with another stream of data that is processed in batches using tools such as Hadoop, Spark, or Hive. The serving layer provides a view of the data using tools such as Cassandra, HBase, or Elasticsearch. The stream-batch join architecture allows for more flexible and scalable processing of the data, such as integrating historical and real-time data, or performing complex aggregations or transformations.

添加您的观点

Matt M.

Revenue Operations Leader | GTM Engineer | Revenue System Architect
举报内容
The stream-batch join architecture merges the strengths of stream + batch processing. Tools like Flink facilitate real-time data processing, making current data readily usable. On the other hand, Hadoop/Spark efficiently manage bulk datasets, extracting deep insights from past data. This architecture is notable for its adaptability, effortlessly blending real-time data with historical context - crucial for anomaly detection. The serving layer, utilizing platforms like Cassandra, provide a consistent view, regardless of the processing method. The ability to perform complex aggregations or transformations on-the-fly while also referencing large batches of historical data offers businesses a competitive edge in their data-driven decisions.

已翻译

赞

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Christelle JULIAS

AI & ML Eng. ? Lecturer ? Advisor
(已编辑)
举报内容
We can't talk about scalable pipelines / workflows / Big Data engineering, without mentioning Big Query, Google's serverless enterprise data warehouse, or Kubeflow, a super tool for deploying ML on Kubernetes. ??

已翻译

赞

Data Engineering

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

Design data pipelines for streaming and real-time data sources?

1

2

3

4

5

6

1 Lambda architecture

2 Kappa architecture

3 Micro-batch architecture

4 Stream-stream join architecture

5 Stream-batch join architecture

6 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

更多Data Engineering相关文章

更多相关阅读内容

Design data pipelines for streaming and real-time data sources?

1

2

3

4

5

6

1 Lambda architecture

2 Kappa architecture

3 Micro-batch architecture

4 Stream-stream join architecture

5 Stream-batch join architecture

6 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

查看其他技能