课程: Advanced Go Programming: Data Structures, Code Architecture, and Testing

Stream processing techniques - Go教程

课程: Advanced Go Programming: Data Structures, Code Architecture, and Testing

开始一个月试用

Stream processing techniques

“

- [Instructor] Now that you understand the fundamentals of what streams are and some of the issues they pose, take a look at two common stream processing techniques, creating append-only logs and windowing. Append-only logs are data structures which are designed to record a series of sequential entries. They're also known as transaction logs or write-ahead logs. Records are added at the end when they are processed and are considered immutable once they are added. They also preserve the order of events, ensuring that the traversal of the list remains consistent. Append-only logs are optimized for quick writes, which makes them ideal for processing streams with quickly occurring values. If a new value is received, then another record is written instead of the old record being updated. The new version of the record can be found and used during the log reconciliation process. Append-only logs are often used by database systems for storing their data. They're also used by event sourcing architectures for capturing and replaying events. Another stream processing technique is windowing. This technique divides a continuous stream of data into smaller subsets called windows. These manageable subsets make it possible for us to run calculations and aggregations instead of processing the entire stream at once. These windows can be split in a few different ways. The first way of splitting data is through non-overlapping windows. Two common ways to split windows are count-based or time-based. Count-based windows produce subsets of N sample size but unequal time durations. Time-based windows produce subsets of the same time duration of N seconds, but unequal sample size. Sliding windows are a variation of both time and count-based windows with the key difference that they allow overlapping data subsets. Sliding windows change their starting point as new data points arrive, but maintain their length, either time-based or count-based. This technique allows us to aggregate historical data together with incoming data, slowly moving preference towards new data as historical data is phased out. As is common in most algorithm problems, windowing is all about breaking the problem into manageable sub-problems. In the case of streams, the amount of data is the biggest challenge. So the purpose of windowing is to break down the data sets into manageable subsets. We will explore this technique a little later in the course.

内容