(2/3) Message Patterns in Enterprise Integration Patterns (EIP) - Advanced Patterns

In the previous article, we explored four foundational patterns introduced in the book Enterprise Integration Patterns (EIP) by Gregor Hohpe and Bobby Woolf. These patterns—Message Channel, Message Router, Message Translator, and Message Filter—form the backbone of many integration solutions.

1. Message Channels
2. Message Filters
3. Message Translators
4. Message Filters
5. Message Splitter
6. Message Aggregator
7. Message Resequencer
8. Composed Message Processor
9. Dead Letter Channel
10. Message Broker
11. Process Manager

In this second article, we will dive into four more advanced patterns: Splitter, Aggregator, Resequencer, and Composed Message Processor. These patterns are designed to address complex messaging scenarios, where messages need to be broken down, processed in parts, and recombined in different ways.

1. Message Splitter

A Message Splitter pattern divides a single message into multiple smaller messages, each of which can be processed independently. This is useful when a message contains multiple items or elements that require separate handling by downstream components. By breaking down a complex message into manageable parts, the Splitter pattern improves flexibility, allowing different services to process individual segments concurrently.

Modern Use Case: In modern cloud-native environments, this pattern is commonly used in data processing pipelines. For example, AWS Step Functions allow defining workflows where batched data is split into individual items, enabling parallel processing of each element. Apache Camel also provides a built-in Splitter component that can split messages based on criteria like list size or specific message attributes.

Example Scenario

Imagine an e-commerce system where an order message contains a list of products. Each product requires different services to handle inventory checks, pricing calculations, and shipping cost estimates. Instead of processing the entire order as a single unit, the Splitter pattern breaks the order into individual product messages, so that each service receives only the relevant product data.

Flow:

1. Producer: The Order Service publishes an order message containing multiple products.
2. Splitter: The Splitter component consumes the order message and splits it into separate product messages.
3. Downstream Consumers:

• The Inventory Service checks the stock availability for each product.
• The Pricing Service calculates the price for each product.
• The Shipping Service estimates the shipping cost for each product.

This approach scales each service independently, processes messages in parallel, and handles individual products with specialized logic.

2. Message Aggregator

An Message Aggregator pattern combines multiple related messages into a single, aggregated message. This pattern is essential when partial results from different sources need to be collected and consolidated before further processing. By waiting for and merging multiple messages, the Aggregator guarantees that downstream components receive complete information.

Modern Use Case: Event-driven architectures frequently use the Aggregator pattern in scenarios where multiple services contribute to a single result. For example, in Apache Kafka Streams, aggregations are common when processing windowed events. Similarly, AWS Step Functions can implement the Aggregator pattern by collecting the outputs of parallel branches before proceeding to the next step. Integration platforms like Apache Camel and MuleSoft also provide built-in support for message aggregation based on custom correlation criteria.

Example Scenario

In an e-commerce system, an order fulfillment process might involve several services: Inventory, Pricing, and Shipping. Each service handles its part of the process independently, but the final order confirmation requires all the information to be combined.

Flow:

1. Producer: The Order Service sends an order message to initiate the fulfillment process.
2. Aggregator: The Aggregator collects responses from all services and combines them into a final order confirmation message.
3. Consumer: The Order Confirmation Service receives the aggregated message and presents the complete order details to the customer.

This approach guarantees that customers receive accurate and complete order details, improving the overall reliability and consistency of the e-commerce platform.

3. Message Resequencer

The Message Resequencer pattern provides that a set of messages is processed in the correct order. This is important when messages arrive out of sequence due to network delays or parallel processing. By reordering messages before passing them to downstream components, the Resequencer helps maintain data integrity and consistency in systems where order matters.

Modern Use Case: In distributed systems, message ordering can be disrupted due to asynchronous processing and network delays. Message brokers like Apache Kafka and RabbitMQ support ordered message delivery, but in cases where order cannot be guaranteed, a Resequencer may be necessary. Apache Camel provides a built-in Resequencer component that can reorder messages based on specific criteria, such as timestamps or sequence IDs.

Example Scenario

In an e-commerce system, consider a scenario where multiple events related to inventory restocking are processed. For example, if products are removed from stock (due to an order) and later restocked (due to a return), processing these events out of order could lead to incorrect stock levels being displayed or recorded. The Resequencer makes sure that stock adjustment events are processed in the correct order, preventing issues such as overselling or stock mismatches.

Flow:

1. Producer: The Inventory Service sends stock adjustment events (e.g., stock decrement for an order and stock increment for a return) asynchronously.
2. Resequencer: The Resequencer consumes the events, identifies their correct sequence based on timestamps or sequence numbers, and reorders them accordingly.
3. Consumer: The Inventory Management System receives the reordered events and updates the stock levels in the correct sequence.

This approach prevents inconsistencies in stock levels, so that customers see accurate product availability and the system remains reliable.

4. Composed Message Processor

The Composed Message Processor pattern allows multiple processing steps to be combined into a single, logical unit. This pattern is useful when a message requires several transformations or operations before reaching its final state. By encapsulating a sequence of processing steps, the Composed Message Processor simplifies the overall workflow and reduces coupling between individual components.

This may seem very similar to the Aggregator pattern, but their purposes and message handling are different. While the Aggregator collects multiple related messages and combines them into a single cohesive message, the Composed Message Processor operates on a single message, applying a sequence of processing steps to transform or enhance it. The Aggregator pattern works on gathering and and unifying messages, such as responses from various services. The Composed Message Processor is ideal for multiple stages of processing, such as validation, enrichment, and formatting.

Modern Use Case: This pattern is commonly used in data processing pipelines where a message must undergo multiple transformations. Integration platforms like Apache Camel and MuleSoft enable chaining of processors to handle complex workflows. AWS Lambda functions can also be composed in a sequence using Step Functions, where each function performs a specific transformation or operation.

Example Scenario

In an e-commerce system, consider a scenario where an order message needs to go through several stages of processing: validation, enrichment, and formatting. These stages are critical for making sure that the order is complete, accurate, and ready for downstream systems like billing and shipping.

Flow:

1. Producer: The Order Service sends a raw order message.
2. Composed Message Processor: The processor consists of three steps:

• Validation: Guarantees the order contains all necessary information (e.g., customer details, payment method).
• Enrichment: Adds additional information, such as shipping costs or promotional discounts.
• Formatting: Converts the order into a standardized format expected by downstream services.

1. Consumer: The Billing and Shipping Services receive the fully processed and formatted order message.

By encapsulating multiple steps within a single Composed Message Processor, the system makes it so that the order is consistently processed and prepared for further actions. This approach improves maintainability and scalability, as new steps can be added or existing ones modified without affecting the overall workflow.

Conclusion

The advanced patterns discussed in this article—Splitter, Aggregator, Resequencer, and Composed Message Processor—address common challenges in distributed systems, such as handling complex messages, providing ordered processing, and composing multiple steps into a cohesive flow.

In the next article, we will cover the remaining patterns introduced in Enterprise Integration Patterns, including Dead Letter Channel, Message Broker, and Process Manager. Understanding and applying these advanced patterns will further improve your ability to design integration solutions.

Stay tuned for the final part of this series!