Fluency Platform. Beyond Pipeline Observability (Part 2)

Data pipes are an evolution in big data operations, addressing challenges in collecting and processing data from distributed sources. Parsing inaccuracies, complexities of shared pipelines, hanging processes, and flow congestion are common issues in big data processing. This paper explores the evolution of data collection methods and highlights persistent challenges, paving the way to understand how data pipes improve big data operations.?

In this paper, it is time to dig deeper into the basic beauty of pipes. Looking at the critical path, it seems simple at first. Pipes move data from one location to another.? If you take a step back and ask, “What was there before pipes?” Before we had pipes, systems, like SIEMs and APM, would use a ‘connector’ to collect data and enter it into the system.??

There are two things that are occurring when we collect data:?

• We are moving the data. We call these features Flow Control. This routes the data and looks at the health, such as congestion and stoppage.?
• We are processing the data so it can be used by the collection system. We refer to these functions as Data Processing, and this performs are parsing, and produces our metrics and analytics.??

Prior to pipes, we had little insight into what the ‘collector` was doing. We managed the collection by looking at the results. If there was no data, we needed to investigate to see if there was a problem with the source or the collector. When the data was late, we guessed there was not enough processing power and added more. Lastly, we had to just have faith that the collector was processing the data correctly and that we would see errors in the results if there were errors.??

In short, we had a lot of faith that the collection was working correctly. We had little insight to validate its operations or to correct problems if there were any. Observable pipes address this shortcoming and provide us with a platform to perform more effective data metrics and processing.?

Pain without Pipes?

As data collection methods evolved over time, the emergence of data pipes addressed longstanding issues encountered in processing information from distributed sources. Data pipes not only streamline the intricate journey of data from source to processor but also provide effective solutions to historical challenges, a shift to better efficiency and reliability in big data management.?

A. Parsing Correctly: One of the primary challenges in big data processing was ensuring accurate parsing of data. Data pipes tackle this issue by allowing sophisticated parsing mechanisms. With built-in intelligence, pipes can accurately interpret and transform raw data into a structured format, ensuring precision in data processing.??

B. Sharing a Pipe: The complexity of sharing data sources among multiple users or systems has been elegantly addressed. Data pipes are designed to facilitate seamless collaboration, providing mechanisms for controlled access, data integrity, and conflict resolution. Once a record is parsed, a pipe can use attributes of the data to determine how to route record. This is often used for its short-term benefit of cost savings, by sending fewer essential data to cheaper storage, or sometimes not storing it at all.?

C. Hanging Processes: Unresponsive or hanging processes, a concern in big data processing, finds a resolution in the architecture of data pipes. Because the pipe sees both the transmitted record and its header, there is a clear distinction on if the issue resides in the data or in the transmission. Pipes can better notify when there is an issue.?

D. Congestion: Congestion within pipes due to the surge in data volume is effectively mitigated by the design principles of data pipes. Through efficient flow control mechanisms, data pipes manage the seamless movement of information, preventing bottlenecks and ensuring that data flows without undue delays. This enhances overall system performance and contributes to the stability of data transmission. Congestion is a common issue when multiple steps are involved. Visibility into the queuing between steps provides immediate understanding of where congestion is.?

In essence, data pipes serve as a comprehensive solution to the historical challenges of data processing. By addressing issues related to parsing accuracy, collaborative usage, hanging processes, and congestion, data pipes have become pivotal components in modern data management systems. The subsequent sections will delve deeper into the specific features and functionalities of data pipes that contribute to their effectiveness in resolving these historical challenges.?

The Pipe Stack?

Pipes share similarities with communication stacks, such as TCP/IP, where data from a source is transmitted to a process or data store. Two primary functions occur within this process: the transmission aspect, like IP and TCP headers, which regulate data flow, and the data processing aspect, analogous to the presentation layer (HTTP) and application layer (HTML) handling the sent data.?

Much like a TCP/IP stack, a pipe stack serves different functions based on whether it involves the movement or presentation of data. Terminology like "header" and "data," and "protocol" and "message" aptly applies in this context. As a result, a pipe encompasses two essential components: Flow Control and Processing. It's noteworthy that flow control serves distinct objectives compared to data processing.?

Like the guaranteed (TCP) and unguaranteed (UDP) routing in TCP/IP stacks, different capabilities exist at each level of a pipe. Consequently, one company's version of a pipe may not correspond to another's solution. For example, certain pipes are designed to prevent message duplication or ensure the guaranteed delivery of a message. Conversely, other pipe solutions delegate recovery and panic-handling outside the pipe, rendering it unguaranteed, akin to UDP.?

Pipe Flow Control?

The observability of pipes implies that we receive alerts when they are not functioning optimally, and that we are provided the insight to what is causing the issue so it can be remedied. A well-operating pipe seamlessly collects, processes, and outputs data in a scalable manner with minimal delays.??

This efficiency in pipe functionality is closely tied to effective flow control, encompassing the management of stoppages and congestion.?

However, several challenges can disrupt this seamless flow, including:?

• Duplicate Data: Ensuring that data is transmitted only once is a critical aspect of flow control. The prevention of duplicate data transmission is fundamental to maintaining data accuracy and integrity.?
• Spike in Data (Back Pressure): Back pressure, a prominent concern in big data, involves issues such as buffer limitations, CPU profiles, and queue monitoring. Addressing late data transmission is crucial in maintaining the integrity and timely processing of information. Abrupt increases in data volume can strain the pipe's capacity. Effectively managing data spikes is vital to prevent bottlenecks and ensure uninterrupted flow.?
• Lost Data: Data loss can occur due to various reasons, including dropped connections, errors in the processing pipeline, or lack of a viable route for the data. Implementing measures for error recovery is essential. Also, the ability to determine when a record fails to route (a data catch) needs to be part of the design.?
• Recovery: In scenarios where data transmission faces interruptions, a robust recovery mechanism becomes crucial. Efficient recovery strategies ensure that the system can resume normal operation after an unforeseen event.?
• Unequal Pressure: Disparities between data production rates and processing or storage capacities can result in uneven pressure. Managing this imbalance is essential for sustained efficiency in data transmission.?

These challenges play a pivotal role in shaping the essential characteristics that a data pipe must embody.?

• Quality of Service (QoS): Quality of Service emerges as a paramount consideration, surpassing mere cost-saving objectives. A robust pipe aims for a delivery guarantee of exactly one, particularly crucial in scenarios where data duplication becomes necessary during urgent moments. Conversely, managing spikes in data to prevent loss highlights the critical need for effective flow control mechanisms within data pipes.?
• Backpressure Control: Backpressure Control is another vital characteristic. Similar to turning off a faucet when a sink overflows, a data pipe needs a mechanism to slow down the inflow of data when it approaches its consumption limit. This may involve implementing an S3 bucket or introducing delays in API calls. Design flexibility is key, considering that not all data sources can be slowed down. While a robust pipe solution is the objective, pushing a system towards likely failure is a design mistake that needs careful consideration.?
• Scalability: Given that pipes are often deployed in large, distributed environments, Scalability is a primary design consideration. While implementing pipes in controlled environments like labs may be suitable for capability testing, scalability is a distinct challenge. All processing has inherent limitations, and identifying bottlenecks in the process is crucial. Understanding the limitations to the flow ensures that the pipe can handle large-scale data processing without compromising performance.?
• Completeness: The question of completeness addresses a system's ability to detect and handle failures. In an imperfect system, the ability to capture errors and the associated data is crucial. This captures the record and state that caused the error, serving a dual purpose. Firstly, it aids in reconstructing the issue for development and testing of solutions. Secondly, the captured error ensures the correctness of the processing results. Unlike pre-pipe systems where errors were often ignored to maintain operational continuity, data pipes provide a clear message path for errors to be sent as notifications, enabling proactive problem resolution.?

Pipe Data Processing?

Data processing is the other use for pipes. It plays a pivotal role in shaping the functionality and utility of the information being transmitted. This involves both transformation and analysis, each serving distinct purposes to ensure the efficacy of the data pipeline.?

1. Transformation: Transformation occurs when we move data from the source and route to its destination. It is needed minimally to determine attributes used for routing. However, transformation changes data from a message format to a hierarchy (of column-row) key-value pairing. This pairing is used both for the route destination and for the pipe itself to perform analysis and derive metrics.?

a) Parsing Data: Parsing involves converting data from an output format into a format compatible with the big data system, facilitating seamless integration.??

i) Error Checking: Robust error-checking mechanisms are implemented to maintain data integrity, identifying and rectifying errors during the processing stage.?

ii) Formatting for Consistency: Ensuring consistency in data formatting is vital for uniformity and ease of interpretation across the data pipeline.?

iii) Type Casting: Data is type-cast to ensure it is stored in a searchable type, optimizing the efficiency of subsequent retrieval and analysis.?

b) Enrichment: Enrichment involves adding data from other sources, such as API calls or lookup tables. Enrichment as context to the record. There are no JOINs in big data. Also, some references are time sensitive, such as converting a dynamic IP address to a system name. Enrichment acts like a JOIN by adding known data at the time that is not in the record.??

2. Analysis: Analyzing data within pipes involves the extraction of meaningful insights, focusing on two key aspects:?

a) Metrics: Metrics refer to the aggregation of data based on specific keys, utilizing functions to present values consistently within defined time slots. This enables the extraction of quantitative measures from the data for further analysis.?

b) Notifications: Notifications arise when aggregated data or key-value pairings meet predefined criteria, triggering alerts. These criteria can be associated with metrics or specific values, offering a flexible and dynamic means of alerting based on data patterns.?

It's essential to note that metrics and notifications are not mutually exclusive; rather, they complement each other. The ability to set notifications for specific metric values adds a layer of sophistication to the alerting system, allowing for nuanced and context-specific notifications.?

In summary, the data processing stage within pipes is a comprehensive endeavor, involving the transformation of data to ensure compatibility and integrity and the analysis of data to extract valuable insights through metrics and notifications. These processes collectively contribute to the efficiency and efficacy of the entire data pipeline.?

Conclusion: Navigating the Critical Path with Pipes?

This segment concludes the exploration of the critical path introduced in the preceding issue, shedding light on the foundational role of data pipes in modern data management. At its core, the basic pipe serves as a conduit, offering visibility and control over data as it traverses from source to storage and/or processing destinations.?

Crucially, the pipe is not merely a passive transporter; it embodies inherent processing capabilities. This functionality allows for the intricate parsing of data, transforming it into key-value attribute pairings. Parsing serves a dual purpose: it ensures that the data reaching its destination is consistent and validated, enhancing the reliability of the overall data pipeline. Additionally, parsing empowers the pipe to intelligently route and process the data it carries.?

The culmination of a well-designed pipe is a fusion of seamless data movement and profound data insight. Beyond the transportation of data, the pipe yields insights in the form of metrics, analytics, and notifications. Metrics provide a quantitative understanding of the data, analytics contribute nuanced interpretations, and notifications serve as alerts triggered by specific data patterns or criteria.?

In essence, data pipes represent a fundamental evolution in data management, offering a holistic solution that not only facilitates efficient data movement but also unlocks the potential for deep data understanding. As organizations continue to navigate the complexities of contemporary data ecosystems, the role of data pipes emerges as indispensable in ensuring a robust and insightful data management infrastructure.?