Microservices Contracts

Reasoning about data is the hardest part of Microservices, however dealing with Microservices Communication comes thereafter.

Communication requires establishing well defined Contracts, Protocol and Interfaces in order to exchange data and information, favor collaboration and keep consistent state within the system.

Microservice oriented applications are evolutionary by nature, a Microservice, in fact, can change and be deployed several times even within the same day. Upgrading a particular service should not compromise the overall system stability and in any case disrupt Business Continuity. 

A good Microservice implementation must guarantee Evolutionary Compatibility and honoring service Contracts must be high on priority. This implies Contracts to embrace some sort of Evolutionary Mechanism.

Schema Evolution allows updating different components of the system independently, at different time, by maintaining different levels of compatibility across Microservices. Schema Evolution must therefore guarantee the natural and independent evolution of each Microservice without the need of replacing other nearby services. 

There are three common aspects regarding Schema Evolution: 

• Backward Compatibility: supports compatibility with previous versions of the contract;
• Forward Compatibility: supports compatibility with future versions of the contract;
• Full Compatibility: supports both backward and forward compatibility.

Let's assume service B reacts to service A events. Following is an example definition of A1 event as per version 1:

Version: 1

A1 {
  {
    "name": "id",
    "type": "long"
  },
  {
    "name": "attribute1",
    "type": "string"
  },
  {
    "name": "attribute2",
    "type": "string"
  }
}

Let's now assume that business asks for some new features so that service A needs to enhance A1 Schema Definition as follows:

Version: 2

A1 {
  {
    "name": "id",
    "type": "long"
  },
  {
    "name": "attribute1",
    "type": "string"
  },
  {
    "name": "attribute2",
    "type": "string"
  },
  {
    "name": "attribute3",
    "type": "string"
  }
}

Version 2 now introduces a new field "attribute3" of type String.

Backward Compatibility is about being able to read older messages (V1) using a new schema definition (V2). See the following example.

Message

A1.V1(id=1, attribute1="attr1", attribut2="attr2")

Schema

A1.V2 {
  {
    "name": "id",
    "type": "long"
  },
  {
    "name": "attribute1",
    "type": "string"
  },
  {
    "name": "attribute2",
    "type": "string"
  },
  {
    "name": "attribute3",
    "type": "string"
  }
}

The newly defined Contract (V2) is not Backward Compatible with the previous Schema Definition (V1) as V2 expects "attribute3" to be present on messages signed by V1. Attempting to read V1 messages using V2 definitions will therefore break protocol communication.

Forward Compatibility is about being able to read newer messages (V2) using an old Schema Definition (V2).

In this case, the newly defined Contract (V2) is Forward Compatible with previous Schema Definition (V1) as V2 satisfy all V1 pre-conditions and requirements.

Still the newly defined Contract is not Fully Compatible as Backward Compatibility is not guaranteed.

In order to guarantee Full Compatibility, V2 Schema needs to be defined as follows:

Version: 2

A1 {
  {
    "name": "id",
    "type": "long"
  },
  {
    "name": "attribute1",
    "type": "string"
  },
  {
    "name": "attribute2",
    "type": "string"
  },
  {
    "name": "attribute3",
    "type": "string",
    "deault": "NA"
  }
}

By setting a Default Value to "attribute3" we can now guarantee Backward Compatibility. Missing values for "attribute3" will in fact be replaced with the Default Value definition. In this example "NA".

Spotting breaking changes, by just looking at and comparing Schema Definitions, is "relatively trivial". However things are not that "simple" when relying on protocols, such as JSON, which do not foresee any mean for Schema Definition.

A JSON implementation of V2 would require spoiling the Contract as "attribute3" would be considered as Optional. However Optional fields have a different Semantical Meaning from Required fields with a Default Value. This practice sounds more like Patching a service rather then implementing Schema Evolution.

Please also note that Conditionally Required fields have again another Semantical Meaning from Optional fields. Conditionally Required fields must be completed if certain conditions are met. Schema Definition does not typically deal with such cases and the implementation usually takes place at the Service Validation level.

Although JSON is simple and transparent, it is also verbose and heavier to serialize/deserialize. Furthermore the lack of a Schema Definition and Specialized Data Types (e.g. Integer vs Long) make it vulnerable to Protocol Skews.

Performing Additive Changes only to Contracts is typically a saver practice. This is especially important when dealing with JSON. Additive Changes in fact reduce the risk of breaking the protocol as removing fields from a Contract is riskier and it requires coordination and regulation such as Contract Versioning.

Contract Versioning mainly defines three types of Semantic Versioning. Semantic Versioning conveys with the X.Y.Z (Major.Minor.Patch) format, encouraging to explicitly state changes to the Contract interface: 

• Minor Versioning: indicates compatible changes of the Contract. E.g. (V1.0.0 -> V1.1.0 -> V1.2.0, etc.);
• Major Versioning: indicates breaking changes of the Contract. E.g. (V1.2.0 -> V2.0.0);
• Patch Versioning: indicates backward compatible bug fixes. (E.g. V1.0.0 -> V1.0.1).

Contract Versioning is especially useful in Orchestration based Interprocess Communication whereas Microservices communicate via API calls and JSON is typically the best wiring option. In this reagard Semantic Versioning enables two different implementations of the same Microservice to concurrently exist within the same system, guaranteeing a higher degree of business continuity and allowing a graceful migration to newer versions.

Brandon Gillespie has written a very nice article about how to deal with API versioning.

More suitable, Schema Based, Data Interchange Protocols exist in the context of Choreography based Interprocess Communications. For instance Thrift, Protobuf and/or Avro.

Avro, in particular, presents several benefits. Among others, the capability of performing Compatibility Tests. Avro as part of the Hadoop ecosystem is becoming de-facto the standard for Big Data and Data Streaming Applications. This makes Avro extremely desirable for Microservices implementations too.

Schema Based protocols also require a mechanism for propagating schemas across different Microservices, Schema Distribution. The Schema Registry is an effective way to distribute/propagate schemas across Microservices, while Common Shared Libraries couple Microservices by design, preventing them to independently evolve.

The Schema Registry is a Service Repository holding schemas and versions about all Microservices Contracts within the system. The Schema Registry is responsible for: 

• Storing schemas;
• Testing compatibilities between newer and older schema versions;
• Eventually tracking schema utilization.

Guen Shapira shared a great article about the importance of having a Shared Schema Registry in order to implement an effective Schema Management strategy.

Finally, it generally is a good practice wrapping or embedding messages (Commands or Events) within an Envelope message. Envelopes in fact allow to add common metadata to a message. For instance a correlation identifier, IP addresses, timestamps or other useful information.