登录查看更多内容

What's the best streaming data platform for your project?

由人工智能和领英社区提供技术支持

Streaming data is data that is continuously generated and processed in real time, such as web clicks, sensor readings, or social media posts. Streaming data platforms are software tools that enable data engineers to ingest, process, analyze, and deliver streaming data to various applications and systems. But how do you choose the best streaming data platform for your project? In this article, you will learn about some of the key factors and features to consider when comparing different streaming data platforms, and some of the popular options available in the market.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

Nagendra U S

CSE ????? || Aspiring Data Engineer || AI/ML Enthusiast???? || Python ?? || SQL || Open to Collaborative Projects…
Ashwini B S

Senior Data Engineer at Merck KGaA | Big Data | Hadoop | Python | Spark | SQL | Bash | AWS | PySpark
Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!

1 Data Sources and Formats

One of the first things to consider when choosing a streaming data platform is the type and variety of data sources and formats that you need to handle. For example, do you need to ingest data from Kafka, MQTT, or HTTP? Do you need to process data in JSON, Avro, or Parquet? Do you need to handle structured, semi-structured, or unstructured data? Different streaming data platforms have different levels of support and compatibility for different data sources and formats, so you need to make sure that the platform you choose can handle your data requirements.

添加您的观点

Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!
举报内容
There are quite a few things to consider on data sources but the core components revolve around the type of data you are ingesting append-only data like events, transactions, and service logs or upsert data like database logs, state updates and so on. If you're going to process data downstream this is a must! Governance is another core component of success, chosing a solution that can manage schema evolution and data contracts will save you countless hours of operational firefighting from bad schema updates. Garbage in, garbage out.

已翻译

赞
Ankit M.

Staff Software Engineer at ServiceNow | Ex {Yahoo, Salesforce, Oracle}
举报内容
- One important consideration while building out your streaming data platform is to understand who the end users are and how they intend to consume the data/extract business value out of it. - While the source data essentially drives the consumption dynamics, it is also important to understand and define the journey of the data until consumption. - Format selection is important because it defines how quickly and efficiently the data can be stored, transmitted and extracted.

已翻译

赞

2 Data Processing and Analysis

Another important factor to consider is the type and complexity of data processing and analysis that you need to perform on your streaming data. For example, do you need to filter, transform, enrich, or aggregate your data? Do you need to apply machine learning, anomaly detection, or sentiment analysis? Do you need to join, window, or partition your data? Different streaming data platforms have different capabilities and frameworks for data processing and analysis, such as SQL, Python, Java, or Scala. You need to choose a platform that can support your data processing and analysis needs and provide the performance, scalability, and reliability that you expect.

添加您的观点

Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!
举报内容
Most Streaming Data Platforms will offer similar functionality when it comes to processing but the devil is in the details. One of the core elements of choice should obviously be the supported languages, SQL-based platforms are generally geared toward intensive Data Engineering workloads while Python-based ones cater to the needs of Data Science teams. But don't forget about DataOps, if every platform allows you to process and JOIN live, not all of them will let you fail and restart safely when things go wrong. That's often what makes or breaks a streaming project. The provisioning model should also come in to play here: do you want to manually maintain nodes provisioning or are you looking for a serverless solution?

已翻译

赞

3 Data Delivery and Consumption

Another key factor to consider is the type and frequency of data delivery and consumption that you need to enable for your streaming data. For example, do you need to deliver your data to a data warehouse, a data lake, or a dashboard? Do you need to consume your data in real time, near real time, or batch mode? Do you need to provide low latency, high throughput, or fault tolerance? Different streaming data platforms have different options and integrations for data delivery and consumption, such as connectors, APIs, or webhooks. You need to choose a platform that can meet your data delivery and consumption requirements and provide the security, governance, and quality that you require.

添加您的观点

Ashwini B S

Senior Data Engineer at Merck KGaA | Big Data | Hadoop | Python | Spark | SQL | Bash | AWS | PySpark
举报内容
One thing I've found incredibly helpful is understanding the intricate balance between data delivery and consumption. Let's delve into this further with an example that highlights the significance of this balance. Imagine you're developing a real-time e-commerce analytics platform. Your job is to update product recommendations in real-time as users browse, involving collecting and processing user data for relevant suggestions. The data delivery consists of setting up efficient connectors to push user interactions to a streaming platform, like Kafka streams. On the consumption side, the dashboard displaying recommendations must be able to pull and render the updated data with minimal delay.

已翻译

赞

4 Data Platform Architecture and Deployment

Another crucial factor to consider is the architecture and deployment model of the streaming data platform that you choose. For example, do you need a standalone, distributed, or hybrid platform? Do you need a cloud-based, on-premise, or edge-based platform? Do you need a serverless, containerized, or VM-based platform? Different streaming data platforms have different architectures and deployment models that affect their cost, maintenance, and flexibility. You need to choose a platform that can fit your data platform architecture and deployment preferences and provide the scalability, availability, and portability that you need.

添加您的观点

Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!
举报内容
This is all business strategy. You generally already know whether your business leans towards build vs buy and at which level (be it hardware, infrastructure, frameworks, or platform itself). There's no reason to digress from what your existing strategy is and most platforms out there already offer hybrid deployment strategy. Depending on your business's size and nature, security, compliance, and pricing model will determine the rest.

已翻译

赞

5 Popular Streaming Data Platforms

There are many streaming data platforms available in the market, each with its own strengths and weaknesses. Popular ones include Apache Spark, a distributed computing framework for large-scale and complex data processing and analysis; Apache Flink, a distributed streaming data platform designed for high-performance and low-latency streaming data processing; Apache Kafka, a distributed messaging system used for building real-time data pipelines and applications; AWS Kinesis, a cloud-based streaming data platform used for building real-time data applications and analytics; and Google Cloud Dataflow, a cloud-based streaming data platform used for building scalable and reliable data pipelines and applications. All of these platforms can run on various platforms such as Hadoop, Kubernetes, or standalone clusters, and are designed to handle high-volume and high-velocity data streams. Furthermore, some of them are fully managed and integrated with other services such as S3, Lambda, Redshift, BigQuery, Pub/Sub or AI Platform.

添加您的观点

Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!
举报内容
Disclaimer: I am a founder at Popsink, a streaming data plateform. Here are a few streaming data platform vendors that operate solutions that can capture, process, and replicate data as streams. Keep in mind that despite these essentially offering "stream processing", they may do it in widely different ways": Confluent, Timeplus, Materialize, Memphis, Estuary, RisingWave, Decodable, ByteWax, Popsink ...and more.

已翻译

赞

6 How to Choose the Best Streaming Data Platform

Choosing the best streaming data platform for your project depends on your specific data needs, goals, and constraints. There is no one-size-fits-all solution, and you may need to compare and evaluate different platforms based on various criteria, such as functionality, performance, cost, ease of use, and compatibility. You may also need to test and prototype different platforms using your own data and scenarios, and get feedback from your stakeholders and users. Ultimately, the best streaming data platform for your project is the one that can help you achieve your data engineering objectives and deliver value to your business and customers.

添加您的观点

Benjamin Djidi

Co-Founder, CEO @ Popsink | Real-time Data Pipelines!
举报内容
A key component of choosing the best platform is to answer the questions 1) who is this for, and 2) what is the job to be done. The "who" will determine properties like the languages the platform needs to suppor, the level of abstraction the platform should provide, and whether build or buy is the right choice. The "what" will determine features, integrations, performance, budget, and the likes...

已翻译

赞

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Nagendra U S

CSE ????? || Aspiring Data Engineer || AI/ML Enthusiast???? || Python ?? || SQL || Open to Collaborative Projects ????||
举报内容
The best streaming data platform for the project depends on your specific requirements, such as data volume, latency, scalability, and integration needs. Some popular options include Apache Kafka, Apache Pulsar, Amazon Kinesis, and Google Cloud Pub/Sub. Each platform has its own strengths and features, so it's essential to evaluate them based on your project's goals and constraints.

已翻译

赞

Data Engineering

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What's the best streaming data platform for your project?

1

2

3

4

5

6

7

1 Data Sources and Formats

2 Data Processing and Analysis

3 Data Delivery and Consumption

4 Data Platform Architecture and Deployment

5 Popular Streaming Data Platforms

6 How to Choose the Best Streaming Data Platform

7 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

更多Data Engineering相关文章

更多相关阅读内容

What's the best streaming data platform for your project?

1

2

3

4

5

6

7

1 Data Sources and Formats

2 Data Processing and Analysis

3 Data Delivery and Consumption

4 Data Platform Architecture and Deployment

5 Popular Streaming Data Platforms

6 How to Choose the Best Streaming Data Platform

7 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

查看其他技能