课程: Advanced Go Programming: Data Structures, Code Architecture, and Testing

Streams - Go教程

课程: Advanced Go Programming: Data Structures, Code Architecture, and Testing

开始一个月试用

Streams

“

- [Instructor] Now, I'll walk you through data streams and large data sets. Due to their size and changing values, they pose unique challenges to engineers. So let's figure out the best way to handle them in Go. Data streams refer to continuously generated or received values. They're discrete values that are received sequentially with each element carrying information to be processed by our application. Most engineers will have encountered some form of data streams. Some examples of data streams are: sensor data, log files, network packets, user input, or real-time events. They're also common in many industries, including finance, e-commerce, social media, and more. Data streams have two main characteristics. First, they are continuous, potentially infinite flows of data. We can visualize them as ever-flowing rivers of data in one direction. The values are discrete, ordered, and sequential. Therefore, the order in which they are processed must be respected. Often, values will contain a key, like an ID, which will indicate order. Secondly, due to their size, data streams require specialized technologies and programming techniques. As you can imagine, we can't save infinite data sets into memory or aggregate them in any meaningful way. You'll often see data streams mentioned alongside specialized streaming technologies. Some of the most common ones are: Apache Kafka, Google Cloud Pub/Sub, RabbitMQ, and Amazon Kinesis. Each of these technologies has its own complexities and implementation details, but they provide error handling and durability to the messages that they transport so we don't have to. They also take care of connectivity and scalability concerns. If you run into one of these technologies, you'll know that you'll be processing large data streams. The streaming technologies mentioned make it possible for data to be delivered to our application code. But we'll have the problem of processing it. Data stream processing is often paired with serverless functions, which are offered by major public cloud providers. Serverless functions are well-suited for data stream processing because of their ability to scale up quickly, which is required by our data streams. Of course, there are considerations that should be taken when it comes to complexity and cost, but these are outside of the scope of our current discussion. That's all the basics of data streams that you need to know for now. We'll experiment with some techniques for processing them soon.

内容