Setup multiple Kubernetes cluster in a Jiff using vcluster

Setting up a multiple Kubernetes(k8s) cluster could be hard. Especially when you want to set up PoC or a Lab environment to try out various tooling such as monitoring, logging, GitOps setup, etc. There are a couple of options to set up a multi-clusters environment using KinD, KinK(Kubernetes in Kubernetes), etc. However, each has its limitations. These tools are intended to set up development environment local workstations and they are not apt to host team-centric clusters or high capacity clusters.

Recently I came across an easy way to set up the multiple flavors?of k8s clusters without much hassle using vcluster.

vcluster i.e virtual cluster not only helps to set up multiple clusters using few commands, it also helps to maintain the lifecycle of it. We can host multiple clusters in a single Kubernetes cluster and manage them effortlessly. As Kubernetes administrator, it eases the operational activities and provisioning.

Use cases

Generally, Kubernetes are used as container orchestration tools and it's being wide adapted in cloud-native software development. Here are the few major use cases that vcluster will address.

• Development team-centric clusters can be provisioned to each team to build and deploy the application. This approach helps grant autonomous power to the development team.
• Multi-cluster environment - Configure and test products such as GitOps engine, Monitor tool, logging platform, etc which requires the multi-cluster environment to test thoroughly
• Shortlived cluster provisioning - To provision cluster which can be used in CI pipeline to test application integration and deployment.

How it works

Virtual clusters are Kubernetes clusters that run on top of other Kubernetes clusters. Compared to fully separate "real" clusters, virtual clusters do not have their node pools. Instead, they are scheduling workloads inside the underlying cluster while having their control plane.

By default, vclusters run as a single StatefulSet pod that consists of 2 containers:

• Control Plane consists an API server, controller manager, and a connection (or mount) of the data store. By default, vclusters use SQLite as data store and run the API server and controller manager of k3s. We can also use a?different data store, such as etcd, MySQL, or PostgreSQL. It is possible to use another Kubernetes distribution as a backing virtual cluster, such as?k0s or vanilla k8s.
• Syncer: makes a vcluster virtual and works in alternative to the scheduler. Syncer copies the pods that need to be scheduled from the vcluster to the underlying host cluster. Then, the host cluster will schedule the pod and the vcluster will keep the vcluster pod and host cluster pod in sync.

Advantage/Benefits

• Lightweight
• No performance degradation
• Flexibility and ease of provisioning
• No admin privileges required
• Single namespace encapsulation

Deploying virtual cluster

virtual cluster can be provisioned either by using a vcluster CLI or helm chart. Vcluster cli also uses helm to install and configure the cluster in the backend. Install the CLI binary in your operating system by the following instruction. You can also customize the provisioning using the helm values.

1. Create a vcluster

# By default vcluster will connect via port-forwarding when used with --connect argument
vcluster create vcluster-1 -n host-namespace-1 --connect

# OR: Use --expose to create a vcluster with an externally accessible LoadBalancer
vcluster create vcluster-1 -n host-namespace-1 --connect --expose         

2. Connect to a vcluster - To connect to an existing cluster, we can use CLI to setup port-forwarding using the below command. Alternatively, you can set up the API service to use the service type as LoadBalancer. It is also possible to set up an ingress for your API which can be exposed to an external network using a host ingress controller. for more info read the documentation.

vcluster connect vcluster-1 -n host-namespace-1
# To create viewonly kubeconfig
vcluster connect vcluster-1 -n host-namespace-1 --service-account viewer --cluster-role view        

3. Use the vcluster - Use the kubeconfig file which can be generated using vcluster cli and update the environment variable KUBECONFIG so that kubectl can use it.

export KUBECONFIG=./kubeconfig.yaml

# Run any kubectl, helm, etc. command in your vcluster
kubectl get namespace
kubectl get pods -n kube-system
kubectl create namespace demo-nginx
kubectl create deployment nginx-deployment -n demo-nginx --image=nginx
kubectl get pods -n demo-nginx        

4. Delete the vcluster

vcluster delete vcluster-1 -n host-namespace-1        

Conclusion

vcluster is a cost-efficient approach with fast provisioning capabilities. It offers strict isolation of workload on the cluster level and enables cluster-wide permission to users. This tool is great for testing and provisioning autonomous or short-lived clusters to teams. Flexibility to run various types of Kubernetes such as k8s, k3s or k0s makes it more versatile. Reduced overhead on host clusters helps to maintain and manage the clusters easily.