Custom debug profiles - A new cool feature of Kubernetes v 1.31!

You want to run some commands inside the container but it is shell-less and You can't debug effectively. Ever faced this?

Kubernetes v 1.31 release has one cool feature of a custom debug profile that addresses this.

Now you can pass the custom profile to Kubernetes debug command to mount volumes and add environment variables.

To understand in-depth, Let's know kubectl debug command first

How 'kubectl debug' works ?

Ephemeral containers are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image doesn't include debugging utilities, such as with distroless images.

Example debugging using ephemeral containers

You can use the kubectl debug command to add ephemeral containers to a running Pod. First, create a pod for the example:

kubectl run ephemeral-demo --image=registry.k8s.io/pause:3.1 --restart=Never        

The examples in this section use the pause container image because it does not contain debugging utilities, but this method works with all container images.

If you attempt to use kubectl exec to create a shell you will see an error because there is no shell in this container image.

kubectl exec -it ephemeral-demo -- sh
        

OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
        

You can instead add a debugging container using kubectl debug. If you specify the -i/--interactive argument, kubectl will automatically attach to the console of the Ephemeral Container.

kubectl debug -it ephemeral-demo --image=busybox:1.28 --target=ephemeral-demo
        

Defaulting debug container name to debugger-8xzrl.
If you don't see a command prompt, try pressing enter.
/ #        

This command adds a new busybox container and attaches to it. The --target parameter targets the process namespace of another container. It's necessary here because kubectl run does not enable process namespace sharing in the pod it creates.

Limitations

If you want to mount an application configuration or data from volume/configmap to your debugger container (the one with busybox image in above example), There is no option for that as debugger container starts with a predefined profiles only.

Kuberentes v1.31 release solve this problem by introducing custom profile for kubectl debug

How custom profile works ?

Imagine you needed volumeMount (from a configMap or PVC) in your debugging session in busybox, you can define a custom profile as file with the following content:

{
    "volumeMounts": [
        {
            "mountPath": "/etc/app",
            "name": "app-config",
            "readOnly": true
        }
    ]
}        

This profile can now used by running:

KUBECTL_DEBUG_CUSTOM_PROFILE=true kubectl debug -it <POD_NAMe> --image=<DEBUG_CONTAINER_IMAGE> --target=<TARGET_CONTAINER> --custom="<PATH_TO_CUSTOM_DEBUG_PROFILE_FROM_ABOVE>"         

This will start an ephemeral swiss-army-knifed debug container beside your minimal/distroless based application container.

Here is another example that modifies multiple fields at once (change port number, add resource limits, modify environment variable) in JSON:

{
  "ports": [
    {
      "containerPort": 80
    }
  ],
  "resources": {
    "limits": {
      "cpu": "0.5",
      "memory": "512Mi"
    },
    "requests": {
      "cpu": "0.2",
      "memory": "256Mi"
    }
  },
  "env": [
    {
      "name": "REQUIRED_ENV_VAR",
      "value": "value2"
    }
  ]
}        

Conclusion

Custom profiling is a new functionality available under --custom flag, introduced in kubectl debug to provide extensibility. It expects partial Container spec in either YAML or JSON format. By allowing the mounting of data volumes and other resources within the debug container, this enhancement provides a significant security benefit for most organizations, encouraging the adoption of more secure, shell-less base images without sacrificing debugging capabilities.