Microservices: A Blueprint Architecture

Recently I wrote a couple of articles about Microservices Integration, Part 1 and Part 2 where I introduced some of the challenges related to Data Management in Microservice based applications as well as some of the patterns and techniques used to tackle these challenges.

Most of these techniques are nowadays considered good practices and industry standards, for instance the Saga pattern. Some others like Event Sourcing are not top level architectures although building an entire system on Event Sourcing is not necessarily and Anti-Pattern. Some other ones are not compulsory and you can live without them, this is still the case for Event Sourcing and CQRS too.

Nevertheless one (new) pattern in particular, Snapshot Propagation, caused quite a stir. I received many private feedback about it, some exciting, some skeptic and some other neutrals.

Thanks to everyone for their feedback, regardless if positive, negative or neutrals. Feedbacks are a way for me to learn more and hopefully sharing back with you my learnings.

Let's get back from where we left, the Anatomy of a Microservice. Following is the diagram I proposed:

From the above Diagram let's draw our attention, for the time being, at the following processes:

• Service Adapter: the Service Adapter is the Anti-Corruption & Transformation Layer of a Microservice;
• Service Processor: the Service Processor is the actual Microservice Business Logic, the Writing Side, responsible for generating events, updating the internal state and delegating the dispatching logic to other components;
• Service Dispatcher: the Service Dispatcher is responsible for routing and dispatching events and/or snapshots (if you like the concept) to nearby services.
• Service Projector: the Service Projection is responsible for materializing Projections, Views or Read Models on a repository, the Reading Side.
• Service APIs: the Service APIs is responsible for exposing Read Models over RPC or REST APIs.

While it's fairly easy to understand all the above mentioned components and the role they play within Microservices, The Adapter Service is often an overlooked component and probably it needs a better explanation with further details.

The Adapter Service

Typically an Anti-Corruption Layer appears in a context where there's a need for Integration between Modern Applications and Legacy Systems. This tier, also known as Facade Layer, is responsible for translating requests or messages from Modern Applications to Legacy Systems and vice-versa. The Anti-Corruption Layer allows preventing two systems to be coupled by their respective design, semantic and poor Data Quality.

One of the first things we learned at school about computing is the self-explanatory concept of Garbage In - Garbage Out. No matter how modern an application is, Data Degradation or simply injecting a wrong or malformed message in a topic by mistake may negatively impact the entire system.

Furthermore, as we already learned from previous articles, Microsevices collaborate (Event Collaboration) by reacting to events published by nearby services, like shown the following simple diagram:

The Order Service produces an OrderCreated event, The Reservation Service and the Payment Service both consume and react accordingly to the OrderCreated event. The OrderCreated event triggers different actions depending on the consuming service. Behind the scenes OrderCreated event is translated into ReserveItems command for the Reservation Service and ProcessPayment command for the Payment Service. This transformation is carried out by the Service Adapter via the Event Command Transformation pattern.

Central to the idea of Event Collaboration is that all Microservices will publish events when something business relevant happens inside of them. Other services may subscribe to that event and do something with it. However Event Collaboration is not free from coupling. Event Collaboration uses events as they had the Semantical Meaning of a command and this introduces another form of Requirement Coupling between Microservices. 

For instance, in a typical e-commerce scenario, the Payment Service needs to validate and process payments from the Order Service. The Payment Service may react to OrderCreated events published by the Order Service. Still different issues may arise such as: 

• The organization may need the Payment Service for various reasons and not just when orders are placed by the Order Service. By listening to OrderCreated events rather than ProcessPayment commands, the Payment Service would need to change and be deployed each time new services would need to interact with it.
• The Payment work-flow may vary depending certain business rules, e.g. Payment Method. For instance the Payment Service will need to react to OrderCreated events from the Order Service in case of Credit Card payments. The same rule does not apply for Card on Delivery or Cash on Delivery payment types. In the case of Card on Delivery the Payment service will need react to the CardSwiped event from the Logistic Service. It the case of Cash on Delivery the Payment Service is not involved at all or the process may be handled manually.

The Event Command Transformation pattern is a way to abstract away, core services, such the Payment Service, from any form of undesired Requirement Coupling which may arise by using the Event Collaboration pattern. 

This extra layer is responsible for transforming, according to given rules, an external event into an internal command. For instance the OrderCreated event with payment method Credit Card would be transformed into a ProcessPayment command for the Payment Service, in the same way the CardSwiped event from the Logistic Service would be transformed into a ProcessPayment command for the Payment Service. The Payment Service, regardless the publishing service or the triggered work-flow will always react in face of ProcessPayment commands. 

The Service Adapter acts both as Anti-Corruption Layer and as an Integration / Abstraction Layer between different services or systems.

A Microservice may have zero, one or many, logical or physical it's up to developers, Service Adapters depending on how many other services it needs to collaborate with. Back to our initial example the Order Service needs to have two Adapters as it needs to react to feedback events sent by the Reservation Service (ItemReserved) and the PaymentService (PaymentProcessed).

Pulling it Together

I wrote about many patterns and techniques in these three articles. I gave out some details, sometimes too many, that may have confused or bored some of you. There are so many things that still need explanations, such as Contracts, Envelopes, Discovery, Orchestration, Choreography, Idempotence, Causal Consistency, etc.. Nevertheless I hope you could gain some of the insights that usually are not easy to find around and most importantly I contaminated your desire to learn more and improve.

It's now time to pull things together by providing and example with source code attached to it.

Out of all patterns and techniques I introduced and described, there are three of them, Event Collaboration, Event Command Transformation and Snapshot Propagation, I personally consider the basic elements for designing modern Microservices based applications.

Let's consider a simplified version of a typical e-commerce checkout scenario. For simplicity the checkout scenario consists of three main services: Checkout Service, Order Service and Payment Service. Following is the relative sequence diagram:

Assuming messages will be exchanged over a Message Broker, I'll be using Kafka in our example, you can abstract away tree different type of topics for each service: A topic for Commands, a topic for Events and a topic for Snapshots. Therefore the Checkout Service, for example, would manage a CheckoutCommands topic, a CheckoutEvents topic and a CheckoutSnapshots topic. Actually there is one more topic, the CheckoutErrors topic, however Failure Management, is not in scope for this article.

Let's then see how Checkout and Order Service may interact together given the latest context by looking a the following diagram:

The above diagrams look very much like a Data Flow. Most of Microservices are indeed Data Flows, directed graph of data flowing between operations. This implies moving from a "traditional" Event Driven Architecture to a more modern Streaming Based Event Driven Architecture.

Stream Processing enables the processing of data in motions. The application logic and queries exist continuously, and data flows through them continuously. Streams can be filtered, transformed, joined, disjoined, merged and much more. Stream Processing enables Continuous Queries, that's where I personally think Snapshot Propagation shines. Most importantly Streaming Processing unifies Data Processing, Analytics and Applications.

Apache Kafka, a distributed streaming platform, is one of the best technologies on the market for designing Streaming Based Event Driven Architecture.

Before leaving you with source code, following is a diagram representing a Kafka based Blueprint Architecture for Microservices.

There are so many things I still would like to explain. However I'm leaving you with some source code (although it's still a work in progress and it's not very well documented) which you may find useful.

This is it! For the time being. Please do not hesitate to comment or contact me if you need further details or explanation.