Troubleshooting/ Debugging: Kubernetes Pods

1) Background

1.1) Throughout we will use kubectl command-line utility to interact with K8S.

1.2) RCA (Root Cause Analysis) of failing Pods:

i) Configurational Errors. Like: deployment and services.

Affect: Containers do not start.

ii) Code Errors.

Affect: Post container start up, application code get failed.

2) Troubleshooting

Step-0: Know the debugging commands

// i) This command gives you the list of debugging commands in K8S.

$ kubectl | grep -i -A 10 debugging

Output: ....


// ii) Lists basic commands you can run on K8S resources.

$ kubectl | grep -i -A 5 Basic

Output: ....

// iii) List your operatiuing K8S resources.

$ kubectl api-resources

Output: ....


// iv) List K8S namespaces.

$ kubectl get ns

Output: ....

Step-1: Check Pods status

// Check the Pods status: Running/ Ready

$ kubectl get pods

If its shows:

a) Pending: Pending status ==> Make sure, there is no pods, with this status.

b) ImagePullBackoff: Docker image registry not accessible ==> Make sure the image name is correct, and the registry is accessible.

c) RunContainerError: ConfigMap/Secrets may missing ==> Make sure ConfigMap/Secrets is available.

d) ContainerCreating: Its taking few time to become available ==> Make sure creation process is completed, before start using it.

e) CrashLoopBackOff: Either Liveness check has failed or Image (Docker) has some issue.

May CMD (docker command): Exiting immediately => RESTARTS column has some number x.

f) Pods in running: But not working appropriately.

Step-2: Check Pods Events

// Describe command: It give more info about errors, when container failed to start.

$ kubectl describe <podName>

It may shows: Less resources allocation, like: RAM, CPU, etc. ==> Increase the corresponding resources, and redeployed it.

Step-3: Check Pods Logs

// Check if application is functioning appropriately or not by checking logs.

$ kubectl logs --tail=20 <podName>

Output: Produces logs

$ kubectl logs -f <podName>

Output: <Nothing>: May be it running the newly restarted pod.

$ kubectl logs <podName> --previous
Output: Produces logs

Step-4: Check Inside Pods by executing commands

//Get inside the pods, and Run "sh", "bash", "ash" Directly in the Pods. To get out of pod. <hit exit>.

$ kubectl exec -it <podName> /bin/sh

Step-5: Check Events at Cluster-Level

//K8S Fires EVENTS: [NORMAL, WARNING, ETC] <== whenever ==> State of Resources changes.

// Better to look into its. All events sorted by time. 

$ kubectl get events --sort-by=.metadata.creationTimestamp

// Warnings only

$ kubectl get events --field-selector type=Warning

// Events related to Nodes

$ kubectl get events --field-selector involvedObject.kind=Node

*************** The END *************************