Handling Different Container Runtimes and Configurations in Kubernetes

When transitioning workloads to Kubernetes, some of the following questions often cross our minds:

1. How to satisfy conflicting workload requirements on the same Kubernetes cluster?
2. Is it possible to lift and shift a few existing workloads to run on Kubernetes with phase-wise modernization?
3. What kind of flexibility can be provided to the Operations folks?
4. Is it possible to use different container runtimes on the same Kubernetes cluster?
5. Is it possible to use different configurations for the same container runtime to satisfy different use cases?
6. How can the workload specify its choice of container runtime?

There are multiple solutions to the above questions. However, the simplest solution is to leverage the Kubernetes RuntimeClass feature. This is your silver bullet to handle different container runtimes for different workload requirements as well as different configurations for a specific container runtime.

Reproducing the “Motivation” section behind RuntimeClass feature from the official Kubernetes docs here:

Motivation

You can set a different RuntimeClass between different Pods to provide a balance of performance versus security. For example, if part of your workload deserves a high level of information security assurance, you might choose to schedule those Pods so that they run in a container runtime that uses hardware virtualization. You’d then benefit from the extra isolation of the alternative runtime, at the expense of some additional overhead.

You can also use RuntimeClass to run different Pods with the same container runtime but with different settings.

As you can infer, RuntimeClass was introduced to make it easier to have different container runtimes or configurations in the same Kubernetes cluster. More flexibility and better utilisation of cluster resources.

The basic format of the RuntimeClass is shown below

kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
    name: <runtimeclass-name>
handler: <configuration-name>
overhead:
    podFixed:
        memory: "<>"
        cpu: "<>"
scheduling:
  nodeSelector:
    <key>: <value>        

Key things to note?:

• The handler field points to specific container runtime or configuration to be used. This is the containerd or cri-o configuration on the Kubernetes nodes. Note that when using RuntimeClass with different runtimes or configurations, you need to use either containerd or cri-o
• The overhead field refers to the overhead of running pods utilizing this RuntimeClass and ensuring proper accounting of the overheads in Kubernetes. This is typically used with different container runtimes.?
• The scheduling field can be used to ensure that Pods running with this RuntimeClass are scheduled on the nodes that support it.

Following is an example of a RuntimeClass for Kata container runtime

kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
    name: kata-qemu
handler: kata-qemu
overhead:
    podFixed:
        memory: "160Mi"
        cpu: "250m"
scheduling:
  nodeSelector:
    katacontainers.io/kata-runtime: "true"        

Corresponding handler configuration when using containerd?

[plugins.cri.containerd.runtimes.kata-qemu]
  runtime_type = "io.containerd.kata-qemu.v2"
  privileged_without_host_devices = true  
  [plugins.cri.containerd.runtimes.kata-qemu.options]
    ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"        

See containerd’s config documentation for more details.

Corresponding handler configuration when using cri-o?

[crio.runtime.runtimes.kata]
  runtime_path = "/usr/bin/containerd-shim-kata-v2"
  runtime_type = "vm"
  runtime_root = "/run/vc"
  privileged_without_host_devices = true        

See cri-o’s config documentation for more details.

Likewise, if you want to specify different configurations for specific container runtime, you can create different handler configurations.

As for the workloads, all that is needed is to explicitly specify the RuntimeClass name in the respective manifests as shown in the following example

apiVersion: v1
kind: Pod
metadata:
  name: <app-name> 
spec:
  runtimeClassName: kata-qemu
  containers:
    - name:  ...        

Using RuntimeClass makes it a lot easier to support different runtime requirements from either the Devs or the Ops folks.