Secure inter-micro-service communication with Spring Boot, Kafka, Vault and Kubernetes -- Part 1 : Introduction and architecture

Links

1. Part 1: Intoduction and architecture <-- This article
2. Part 2: Setting up Kubernetes and Kafka
3. Part 3: Setting up Vault
4. Part 4: Building micro-services
5. Part 5: Deployment and testing

Introduction

Micro-services is a design pattern where large monolithic applications are segregated into smaller, more manageable components. These components can work together to solve a particular business problem.

 For that, the components need to talk to each other. Communication between components could be achieved through many ways: RESTful web services, SOAP web services, RPC, messaging, etc. One popular implementation of messaging (publish / subscribe) is Kafka.

In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.

https://kafka.apache.org/

Publish subscribe

Kafka follows the publish subscribe pattern. This pattern works like a bulletin board. For example, if Alice put up an announcement on a bulletin board. Bob and Charles could both read them. They could read them at the same time, or one after the other. Bob could read the board today and Charles could read it tomorrow. The announcement by Alice would remain on the bulletin board until the expiration date for it elapsed.

In a messaging system, we would have publisher (Alice) and subscribers (Bob and Charles). The bulletin board is called a Topic and the announcement is called an Event or a Message.

As you can see, a Topic would need to retain data for a prescribed time period. Because of this, there is a need for the data in the Topic to be secured. Unfortunately Kafka (at the time of this writing) does not handle encryption of data end-to-end.

This is where Vault comes in.

Vault

Vault provides encryption as a service for us. This allows us to have and end to end encryption scheme for our messages.

But why Vault? Do we really need it? Now, of course, all we need to encrypt a message coming from a publisher to a subscriber is a private key/public key infrastructure. It is quite easy to just create these keys and embed them into the Publisher and Subscriber services as per below:

The problem is when we have multiple subscribers, each one with its own set of private / public keys. The problem would come when we need to manage the lifecycle of the keys. When it comes to expiry for example, both keys need to be replaced. If the keys are embedded, then redeployment may be needed.

Another problem is when keys need to be adhoc-ly revoked and replaced. There is no easy way to do it with embedded keys.

The problem becomes more exaggerated in micro-services environment where different services have different sets of keys - for example the service mesh below:

Trying to manage or revoke keys within a merely 4 services mesh is going to be next to impossible.

Vault allow us to centrally manage all these keys:

In addition, Vault allows us to dynamically assign which service has access to which key. If ever, a particular service is compromised, keys could easily be revoked and restored if once the system is secured again.

Architecture

The architecture of our application is presented below. It supports secure end to end communication (messaging) using Vault and Kafka :

Application flow:

1. A Transaction service would initiate a transfer of money from one Deposit Account to another.
2. The Transaction Message is created by the Transaction micro-service and encrypted by Vault.
3. The encrypted Transaction Message is passed to the Request Topic
4. The subscriber will consume the Message and enact the Transaction - transferring money from one account to another.
5. The subscriber will then reply back to the Transaction service with the resulting balance through the Response Topic