Microservices Integration - Part 1

Reasoning about Integration, especially about data integration, is one of the hardest part while designing Microservices oriented applications. This is mainly due the following reasons: 

• Microservices operate in highly distributed environment;
• Microservices are “federated”, they hold their own database, data and data model;
• Developers are used to think in terms strong consistency, transactions, referential integrity and data normalization;
• Data typically is not considered a first class citizen.

Distributed Systems, with regards to data, need to deal with the following challenges: 

• Data Sharing: A service requiring some data from other services,
• Distributed Transactions: Data updates involving two or more services.

Data Sharing involves the followings: 

• Collaboration: a service event which directly or indirectly triggers an action on another service. For instance the OrderValidated event, from the Order Service, which triggers the ProcessPayment action, on the Payment Service.
• Validation: a service which needs to perform some data validations. For instance a Shipping Service which needs to contact the Customer Service in order to validate some customer data such as customer id or email.
• Composition: a service which requires additional information from other services in order to produce some operational* reports or dashboards. For instance the Order Service dashboard that needs to have additional static data such as customer region, country, etc. from the Customer Service.

* Please note that operational dashboards belong to the application domain and not to the analytics domain.

A fundamental activity while designing Microservices based applications is decomposing complex problems into more understandable and manageable pieces. 

Domain Driven Design (DDD) is an approach to software development for complex needs offering a way to create a modular domain model that can be partitioned across services. The fundamental assumption of DDD is that human beings are incapable of building enterprise applications, but are capable of understanding small, well-bounded problems (a "domain"). 

DDD is a perfect tool for modeling Microservices, It helps decomposing complex problems into more understandable and manageable pieces. It is a good approach to software modeling and it is the facto becoming the blueprint for designing Microservice oriented applications. 

DDD comprises of different concepts such as Context, Domain, Model and Ubiquitous Language, strategic design principles such as Bounded Context, Continuous Integration and Context Map as well as several building box such as Entity, Value Object, Aggregate, Domain Event, Service, Repository and Factory. 

Core to DDD is the Bounded Context. A Bounded Context in simple terms can be considered as a small application, containing its own Domain, code and persistence. Bounded Context allows to divide a large application in smaller components (Bounded Context) making the overall system more modular. By splitting a large application in many smaller Bounded Context you can achieve modularity, Separation of Concerns and Loose Coupling. 

For instance a typical e-commerce application is made of different context: Inventory, Shipping, Order Interface, Payment, etc. Each Bounded Context has a specific responsibility and can operate in an quasi-autonomous way.

Bounded Contexts need to interact with one another in order to obtain the information they need. For instance the Payment Service needs to be notified by the Checkout Service or the Order Service about a customer request for checkout. From a technical perspective, different contexts can communicate in different ways, for instance via Remote Procedure Calls (RPC), via RESTfull HTTP calls or via Messaging. 

Context Map is the DDD strategic design principle used to define Contracts between different Contexts. 

Message base communication is probably the best way of integration in Microservices as it guarantees a higher degree of decoupling. For instance it reduces, but not always removes, Temporal Coupling derived by the blocking nature of more traditional, synchronous, implementations like RPC and REST. Message base communication requires, of course, messages as a way to communicate between different systems whereas messages are immutable. Immutable messages are Events. An Event is something that has happened in the past.

Events are typically used to interact with other parties serving mainly two different purposes: 

• Service Collaboration: notifying interested parties that something has happened;
• Service Integration: telling or describing interested parties what exactly happened.

In DDD, a Domain Event is something that has happened in the past, that is of interest to the business. Domain Events allows integrating Bounded Contexts, with minimal coupling and dependencies. 

In traditional DDD implementations, Domain Events are typically used both as a mean for Service Collaboration and Service Integration. Domain Events are very often a carrier of Change Propagation across different context boundaries. However this practice may hide a disruptive force that usually leads to different forms of coupling.

As we already know, we need to stay away as much as we can from coupling as it prevents the system to easily and seamlessly scale and evolve. 

Literature tells us that Microservices are decoupled when they don't need to know anything about each other internal structure. However experience tells us that Microservices have different forms of coupling derived from Business Logic, Protocols and also Data. 

The way Domain Events are currently used in many Microservices implementations, as a mean of Change Propagation may lead to Integration Coupling and more precisely to Requirement and/or Data Coupling.

Requirement Coupling 

Two or more services are coupled by requirement if a service is forced to change its own implementation due to some changes or new requirements of an existing or a new service kicking in. 

For instance, in a typical e-commerce scenario, the Promotion Service, responsible for creating sales promotions, may need to publish certain products as promotional sales each time the reference selling price of a product drops below all competitors prices.

Although such implementation is quite naive for a Promotion Service, it is useful for demonstration purposes. 

The Pricing Service is responsible for calculating the reference selling price, or more simply the selling price, for all products in the catalog, based on some given business rules and inputs such as the product catalog, the inventory, competitors prices and sales history. 

Following is the Pricing Service diagram.

The Pricing Service is already performing some pricing calculation and it's connected to all sources the Promotion Service would need in order to fulfill business requirements. It therefore may seem natural leveraging the Pricing Service in order to accomodate Promotion Service's needs.

For instance the Pricing Service could adapt its own logic in one of the following ways: 

• By enhancing the PriceUpdated Domain Event with extra information such as a isPromotion flag, specifically for the Promotion Service, as illustrated in Promotion Service solution 1;
• By publishing a new custom PromotionUpdated Domain Event in order to fulfill Promotion Service needs, as illustrated in Promotion Service solution 2.

Promotion Service solution 1.

Promotion Service solution 2.

Very often teams turn at each other asking, sometimes begging, for some protocol changes or enhancements in order to fulfill their own requirements. This is typically accomplished by enhancing existing Domain Events with extra data or by adding new custom Domain Events. As a result Domain Events become Fat or carriers of Change Propagation in order to satisfy integration needs.

Both solutions exploit Domain Events and leverage existing infrastructure in order to carry data and fulfill business requirements. Although they may look like a good or reasonable solution, they both lead to Requirement Coupling and as such they are strongly discouraged. In fact any new requirement to the Pricing Service may now impact the Promotion Service, any new requirement to the Promotion Service may require changes to the Pricing Service as well. As multiple services join this game of integration we will eventually end up with a Spaghetti Microservices architecture.

Furthermore, by looking at the solutions proposed we may wonder why we need a Promotion Service at all. The Pricing Service could just act as a Promotion Service as well.

Still the Promotion Service is a different business with its own requirements. In DDD it is a different Bounded Context and therefore it is a different Microservice. The Promotion Service, in fact, is responsible for its own data and logic.

A better solution may therefore look like the following diagram:

Although it may look like a more complicated solution, it decouples services from business logic and therefore from business requirements. Decoupling the pricing calculation logic, specific to the Promotion Service, rather then enhancing the existing one (Pricing Service), in the long term is a better option as it guarantees a higher degree of modularity, flexibility and scalability.

Still changes to any of the services, for instance Sales History, may know impact Pricing and Promotion services. Nevertheless they are know only coupled by data and protocol. This is the reason why Contracts are a very important concept while designing Microservice based applications. However this is material for another article.

Data Coupling

A special form or Requirement coupling is Data Coupling whereas which data should or should not be part of a Domain Event is always source of problems and confusion while designing DDD based Microservices. 

This is especially true when two or more Consumer Services (services that consume messages) rely on a Producer Service (service that produces messages) Domain Events, like shown the image below:

In a typical e-commerce scenario, the personalized Recommendation Service, responsible for recommending products to customers, and the Shipping Service, responsible for managing and tracking delivery, might both rely on the Order Service Domain Events.

The Recommendation Service may want to exclude from the recommended items all products already purchased by a customer. The Shipping Service needs to know about customer purchased products in order to manage and track shipments. Recommendation and Shipping Services may require, and they certainly do, different level of information from the Order Service. The same might be true for the Payment Service, responsible for processing payments upon customer orders, or any other services relying on Order Service's Domain Events.

The Recommendation Service may only need to know about purchased SKUs and associated customer ids, the Shipping Service, instead, needs to have a broader view, requiring information such as quantities, prices, delivery options, payment methods, customer address, customer contacts, etc. 

Typically the Order Service would publish an OrderConfirmed event in order to signal a new customer purchase. Quite often existing Domain Events change in order to accommodate different nearby services needs. This might be the case for the OrderConfirmed domain event which needs to accomodate different services needs, e.g. Recommendation Service, Shipping Service, Payment Service and others. Some other times integration gaps are filled by creating new and custom Domain Events.

How big or small, thin or fat the OrderConfirmed event should be in order to comply with nearby service requirements is always source of problems and confusion. Most importantly it also leads to Data Coupling between services. 

Any future change to the Order Service may impact the Recommendation Service, the Shipping Service or any other neighbour service, preventing the former from scaling autonomously and independently. Constantly changing existing Domain Events or adding new ones, in order to accomodate new services or requirements, have bad consequences in terms of maintenance and evolution of the Order Service and the overall system. 

Conclusion Part 1

Change Propagation and dealing with data is the hardest part while reasoning about Microservices. Microservices, contrary to SOA and Monoliths, don't have a shared data layer as each Microservice has its own data store and data model. 

Fat Events are dangerous and most certainly lead to Integration Coupling. Domain Events should stay thin and any extra information should be accessed by either querying Materialized Views like in a traditional Request/Response context, or by leveraging a new paradigm which I'll name Snapshot Propagation, more suitable in a Publish/Subscribe context. 

Things are not always easy, straight and linear. You need a DDD master in order to spot these issues. I consider DDD being an Art and unfortunately I'm not a DDD artist.

Nevertheless I think I got enough experience with both Microservices and Big Data analytics in order to advice about using Domain Events as a Collaboration pattern and avoiding them as an Integration pattern.

I finally hope you found this article useful and please excuse me for my english or anything that is not clear enough. I'll try to make it more clear in the second part of the article, Microservices Integration - Part 2 where I'll also explain what I mean by Snapshot Propagation and why I think we need a State Transfer Representation "protocol".

Any comment, feedback or constructive criticism is very much appreciated!