How to build a multi-cluster environment and establish a cluster mesh

The following picture illustrates how Calico can set up cluster-mesh connectivity between your clusters.

Calico provides different ways that you can use to build a multi-cluster environment providing flexibility to align with your networking infrastructure and requirements. The best part about Calico is that after joining two clusters together it will automatically provide the cluster mesh and you can start using resources in different clusters by using their internal cluster services and IPs.

Cluster mesh in a flat networking environment

The configuration of a cluster mesh can vary depending on the underlying infrastructure, but its fundamental purpose remains the same and in essence, it is deployed to establish connectivity between standalone clusters.

As an example, in an environment where participating nodes of our clusters are connected directly via a broadcast domain, we can quickly establish a mesh with a routing protocol that propagates internal routes to external entities. This will allow us to advertise internal routes without exposing them via a nodeport service to our intended services.

The following image illustrates a common design for flat network:

Consider viewing our video, BGP for Kubernetes with Calico open-source, for a comprehensive tutorial on deploying a cluster mesh using the full-mesh method.

Cluster mesh in an enterprise or cloud networking environment

In a complex networking environment such as the cloud or an enterprise network, underlying infrastructure resources that form the cluster are usually separated into individual broadcast domains. This limitation arises because each entity within these domains is mandated to traverse through a gateway to reach its intended destination. However, by default, this additional hop (the gateway) needs to gain awareness of the internal cluster resources that we create within our Kubernetes clusters. Consequently, the gateway drops packets bound for these internal resources, rendering simple routing methods inadequate.

Now that we know about the problem, let’s review how Calico can be utilized to solve such an issue.

In order to establish a cluster mesh in such an environment we can use two methods:

• Top of the rack (TOR)
• Overlay

TOR method (Recommended)

In an enterprise or cloud environment, resources are typically interconnected through an intermediary gateway. We usually recommend TOR to our customers since building a route propagation mechanism with the cloud provider automatically makes your cluster aware of all cloud resources that are under your supervision. This also plays a role in your cluster being highly available via the underlying network infrastructure that cloud providers.

Suppose you can configure the cloud gateway and pair it with a robust Container Network Interface (CNI) like Calico. In that case, you can utilize BGP routing to establish a cluster mesh. Calico BGP integration offers dynamic routing, allowing you to propagate internal pod and cluster IP routes to other resources in your network infrastructure.

The following image illustrates a common design for a BGP capable cloud environment:

In many scenarios, the TOR (Top of Rack) enhances high availability and fault tolerance. This approach optimizes network traffic distribution, mitigates single points of failure, and further fortifies the reliability and resilience of your infrastructure.

Note: View this tutorial for a comprehensive guide on deploying a cluster mesh using the TOR method.

IPIP overlay

In some cases, you might not be able to access your gateway to modify its settings, or it might not support BGP peering. In such a scenario, you can use an IPIP overlay to encapsulate the traffic to your destination cluster. An overlay network allows network devices to communicate across an underlying network (referred to as the underlay) without the underlay network having any knowledge of the devices connected to the overlay network.

Read the full article What is a Kubernetes cluster mesh and what are the benefits?