The Ephemeral Containers in Kubernetes
Debugging in a Kubernetes ecosystem is essential for DevOps professionals. However, certain constraints, such as the immutable nature of a Pod’s specification, present challenges.
Deconstructing Debugging Dilemmas
Shipping production container images with a complete Linux environment and debugging tools is neither efficient nor secure. Using kubectl cp to inject debugging tools to run containers might seem like a solution, but it's not always viable. Even if debugging tools are available, the kubectl exec command might be insufficient, particularly if the container is constantly crashing.
Exploring Alternate Debugging Approaches
How does one address debugging in the face of immutable Pod specifications?
Direct debugging from a cluster node is an option, but SSH access may be restricted. This leaves few alternatives. However, there’s a potential game-changer: tweaking the immutability constraints of the Pod.
The Ephemeral Container Revelation
Imagine integrating transient, constrained containers into an active Pod without requiring a restart. This added container could inspect its peer containers in a malfunctioning Pod, regardless of their state or content.
Welcome to the world of the Ephemeral Container. Described as a specialized container momentarily added to an extant Pod for user-driven tasks like troubleshooting. The theory behind it is intriguing, prompting a deeper dive into its practicality.
Kubectl debug Command
Kubernetes has introduced support for ephemeral containers, further enriching its capabilities. This addition comes by augmenting the Pod specification with a novel attribute aptly named ephemeralContainers. Notably, this attribute encompasses a series of Container-like objects . It distinguishes itself by being among the limited set of Pod spec attributes that can be altered after the creation of a Pod instance.
Incorporating elements into the ephemeralContainers list facilitates the initiation of new containers within an existing Pod. The properties of the EphemeralContainer specification are extensive and adjustable.
However, it’s worth noting that the larger fraction of Kubernetes users may not interact directly with the ephemeral containers API. This is primarily due to the robust kubectl debug command, designed to encapsulate ephemeral containers within a more user-friendly debugging workflow. Let's delve deeper into how the kubectl debug command can be optimally utilized.
Using kubectl debug
$ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: slim
spec:
selector:
matchLabels:
app: slim
template:
metadata:
labels:
app: slim
spec:
containers:
- name: app
image: gcr.io/distroless/python3-debian11
command:
- python
- -m
- http.server
- '8080'
- name: sidecar
image: gcr.io/distroless/nodejs-debian11
command:
- /nodejs/bin/node
- -e
- 'setTimeout(() => console.log("done"), 999999)'
EOF
$ POD_NAME=$(kubectl get pods -l app=slim -o jsonpath='{.items[0].metadata.name}')
In the event that a particular Deployment exhibits unexpected behavior, one may find limited utility in the kubectl exec command. This is primarily due to the fact that distroless images frequently omit foundational diagnostic tools.
To further elucidate this point, let’s proceed by examining Pods through the use of an ephemeral container:
$ kubectl debug -it --attach=false -c debugger --image=busybox ${POD_NAME}
The aforementioned command integrates a new ephemeral container named debugger to the designated Pod, utilizing the busybox:latest image.
Below is an analysis of how this command influences the Pod specification:
$ kubectl get pod ${POD_NAME} \\
-o jsonpath='{.spec.ephemeralContainers}' \\
| python3 -m json.tool
[
{
"image": "busybox",
"imagePullPolicy": "Always",
"name": "debugger",
"resources": {},
"stdin": true,
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"tty": true
}
]
and on its status:
$ kubectl get pod ${POD_NAME} \\
-o jsonpath='{.status.ephemeralContainerStatuses}' \\
| python3 -m json.tool
[
{
"containerID": "containerd://049d76...",
"image": "docker.io/library/busybox:latest",
"imageID": "docker.io/library/busybox@sha256:ebadf8...",
"lastState": {},
"name": "debugger",
"ready": false,
"restartCount": 0,
"state": {
"running": {
"startedAt": "2022-05-29T13:41:04Z"
}
}
}
]
If the ephemeral container is running, we can try attaching to it:
$ kubectl attach -it -c debugger ${POD_NAME}
Connecting to localhost:8080 (127.0.0.1:8080)
writing to stdout
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "<https://www.w3.org/TR/html4/strict.dtd>">
<html>
...
</html>
But there is actually a problem:
领英推荐
$# ps auxf
PID USER TIME COMMAND
1 root 0:00 sh
14 root 0:00 ps auxf
In examining the output of the ps command within the debugger container, it becomes evident that it exclusively displays the processes of that particular container. When utilizing kubectl debug, it appears to primarily provide a shared network namespace—potentially an IPC (Inter-Process Communication) namespace as well. This suggests that it might operate under the same parent cgroup as the other containers within the Pod. From a professional standpoint, such a configuration seems rather restrictive, potentially hindering a comprehensive debugging experience.
Using kubectl debug with a shared pid namespace
In a thorough examination of the official Kubernetes documentation , one can identify explicit mentions of this particular issue. The documentation recommends a solution: the activation of a shared pid namespace for every container within a Pod.
To implement this, one should set the shareProcessNamespace attribute within the Pod template specification to true.
$ kubectl patch deployment slim --patch '
spec:
template:
spec:
shareProcessNamespace: true'
$ kubectl debug -it -c debugger --image=busybox \\
$(kubectl get pods -l app=slim -o jsonpath='{.items[0].metadata.name}')
$## ps auxf
PID USER TIME COMMAND
1 65535 0:00 /pause
7 root 0:00 python -m http.server 8080
19 root 0:00 /nodejs/bin/node -e setTimeout(() => console.log("done"), 999999)
37 root 0:00 sh
49 root 0:00 ps auxf
The improvements are evident. This mirrors the debugging experience one would expect on a standard server, where all operational processes are consolidated and visible in a singular interface.
Within the Pod specifications, alterations to the shareProcessNamespace attribute inevitably results in the re-creation of the Pod. Notably, when this adjustment is made at the Deployment's template specification level, it triggers an extensive rollout. Such a process can present significant operational overhead, particularly when one is merely attempting to debug. These containers were designed to be seamlessly integrated without causing any disruptions to the existing Pod.
Hence, unless your specific workloads are configured with the shareProcessNamespace enabled by default (a practice that arguably compromises the isolation among containers), the industry is still in pursuit of a more refined solution.
Using kubectl debug targeting a single container
In container orchestration, it’s important to note that a singular container can only exist within one pid namespace at any given moment. Should a pod neglect to initialize with a shared pid namespace from inception, this would preclude the ability to launch an ephemeral container with visibility into the processes of all the associated containers.
That being said, from a technical standpoint, one can still instantiate a container that leverages the pid namespace of a specific container within a pod.
$ POD_NAME=$(kubectl get pods -l app=slim -o jsonpath='{.items[0].metadata.name}')
$ kubectl debug -it -c debugger --target=app --image=busybox ${POD_NAME}$# ps auxf
PID USER TIME COMMAND
1 root 0:00 python -m http.server 8080
13 root 0:00 sh
25 root 0:00 ps auxf
Conclusion
Bulky container images for Kubernetes workloads spell inefficiency and elevated security risks. The prolonged durations of CI/CD pipelines are indicative of such inefficiencies, and the surplus components in these images enhance susceptibility to threats. Hence, dynamically integrating debugging tools into malfunctioning Pods becomes pivotal. Ephemeral containers in Kubernetes fulfill this need seamlessly, fortifying our clusters.
My exploration of the revamped kubectl debug the command has been illuminating. However, harnessing its full prowess necessitates a profound grasp of container mechanics and Kubernetes nuances. Lacking this understanding can lead to unexpected behaviors, from missing processes to unintended widespread Pod restarts.
Resources