The Microservice Takeover: “How low-code is modernizing applications”

In my previous post, we explored the advantages of event-driven architecture (EDA) and the pub/sub messaging pattern for building modern, scalable systems. If you haven't already, I recommend checking out that post for a deeper dive into this powerful messaging pattern.

Publish-Subscribe (pub/sub) for Asynchronous Communication

In this article, we'll dive deeper into the practical applications of the microservice approach with a real use case, focusing on the pub/sub messaging pattern for asynchronous communication.?

Scenario: “Monolithic Legacy Application”

XYZ Enterprises, a large company, relies on a monolithic legacy application for various business functions. This application follows a linear workflow:

1. "Order Service" triggers the process.?
2. The system sequentially performs the following actions:

This approach presents several challenges:

• Limited scalability: Scaling a single service necessitates scaling the entire integration.
• Tight coupling: Updating one service might break others due to tight dependency between components.
• Reduced fault tolerance: A single service going down can disrupt the entire workflow.?

Solution: “The Pub/Sub Application”

Pub/Sub addresses the previously stated challenges by enabling vertical or horizontal scalability for individual consumers, decoupling services to reduce dependencies, and enhancing fault tolerance by isolating failures to specific functionalities.?

In this microservice approach, when an order is made, the Order service publishes an event that is routed to multiple consumers, allowing each service to act autonomously based on its logic, diverging from linear flow.

This diagram showcases the solution:

This means that whenever an order is made, it will:

• Publish an event to the event broker.
• The event broker will route the message to the consumers subscribed to that event and will act accordingly.

This establishes a complete decoupling between each service, and because of the nature of the pub/sub pattern, each microservice can scale and fail independently. Additionally, since there are no single points of failure, error cascading is mitigated.

Plan for Failure

Even in robust architectures, unforeseen errors can arise. To address these errors, we will now explore how to implement a reprocessing pipeline to manage transient errors, preventing them from disrupting the overall workflow.

Consider a scenario where the "Order service” triggers an event of an order made. The flow will proceed as follows:

1. The "Inventory service" receives the event.
2. Due to a temporary network glitch, the service encounters an error while updating the inventory.
3. Within the “Inventory service”, errors are sorted by their respective status code to separate client errors from server errors.
4. Records associated with client errors will deliberately not be reprocessed (out of this article’s scope).
5. Records associated with server errors will temporarily be stored in a database for future reprocessing. These records may include pipeline details (such as name), reprocessing parameters (like retry number), interval time between reprocessing attempts, as well as the payload and the specific error encountered.

The Reprocessing Pipeline for Transient Errors

This process involves the following steps:

1. The reprocessing pipeline starts by querying the database containing the error records stored from the “Inventory” service.
2. For each retrieved record, the pipeline checks the number of retries associated with that specific record. At this point, a logic will be implemented that will only try to reprocess records that haven't surpassed a predefined threshold (e.g., maximum of 3 retries).
3. If an error record hasn't exceeded the maximum retry limit, the reprocessing pipeline increments the "attempt" count field by 1 and republishes the record back to the processing pipeline (the "Inventory service"). This triggers another attempt to update the inventory .
4. If an error record has exceeded the maximum retry threshold, the reprocessing pipeline can take various actions depending on the functional requirements.

Why Retries Matter

Imagine a scenario where an order is placed, therefore, triggering a sequence of events. However, due to a network glitch, the "Inventory service" encounters an error while updating the stock levels. In this context, the reprocessing pipeline (functioning periodically as a scheduler), retrieves the failed event(s) for reprocessing.

However, if a system is already overloaded and experiencing timeouts, retries may exacerbate the situation. Each retry adds to the system's burden potentially causing more issues.

To mitigate these challenges, we can adopt various approaches. For example, initiating retries only when the dependent system is healthy and likely to respond successfully. This opens up a new topic, which is how to integrate the Circuit Breaker pattern for error handling into your workflows. This topic will be covered in upcoming articles.

Conclusion

Pub/Sub's role in modern architecture includes multiple benefits as already mentioned. Primarily, it decouples publishers and subscribers, enabling asynchronous and fault-tolerant message handling. Reprocessing strategies enhance this messaging pattern by retrying failed messages due to transient errors, improving system reliability.

Thank you for reading and keep an eye out for upcoming articles!