Kubernetes Troubleshooting

Cluster Management

Display endpoint information about the master and services in the cluster

kubectl cluster-info        

Display the Kubernetes version running on the client and server

kubectl version        

Get the configuration of the cluster

kubectl config view        

List the API resources that are available

kubectl api-resources        

List the API versions that are available

kubectl api-versions        

List everything

kubectl get all --all-namespaces        

Daemonsets

Shortcode = ds

List one or more daemonsets

kubectl get daemonset        

Edit and update the definition of one or more daemonset

kubectl edit daemonset <daemonset_name>        

Delete a daemonset

kubectl delete daemonset <daemonset_name>        

Create a new daemonset

kubectl create daemonset <daemonset_name>        

Manage the rollout of a daemonset

kubectl rollout daemonset        

Display the detailed state of daemonsets within a namespace

kubectl describe ds <daemonset_name> -n <namespace_name>        

Deployments

Shortcode = deploy

List one or more deployments

kubectl get deployment        

Display the detailed state of one or more deployments

kubectl describe deployment <deployment_name>        

Edit and update the definition of one or more deployment on the server

kubectl edit deployment <deployment_name>        

Create one a new deployment

kubectl create deployment <deployment_name        

Delete deployments

kubectl delete deployment <deployment_name>        

See the rollout status of a deployment

kubectl rollout status deployment <deployment_name>        

Logs

Print the logs for a pod

kubectl logs <pod_name>        

Print the logs for the last hour for a pod

kubectl logs --since=1h <pod_name>        

Get the most recent 20 lines of logs

kubectl logs --tail=20 <pod_name>        

Get logs from a service and optionally select which container

kubectl logs -f <service_name> [-c <$container>]        

Print the logs for a pod and follow new logs

kubectl logs -f <pod_name>        

Print the logs for a container in a pod

kubectl logs -c <container_name> <pod_name>        

Output the logs for a pod into a file named ‘pod.log’

kubectl logs <pod_name> pod.log        

View the logs for a previously failed pod

kubectl logs --previous <pod_name>        

For logs we also recommend using a tool developed by Johan Haleby called Kubetail. This is a bash script that will allow you to get logs from multiple pods simultaneously. You can learn more about it at its?Github repository?Here are some sample commands using Kubetail.

Get logs for all pods named with pod_prefix

kubetail <pod_prefix>        

Include the most recent 5 minutes of logs

kubetail <pod_prefix> -s 5m        

Manifest Files.

Another option for modifying objects is through Manifest Files. We highly recommend using this method. It is done by using yaml files with all the necessary options for objects configured. We have our yaml files stored in a git repository, so we can track changes and streamline changes.

Apply a configuration to an object by filename or stdin. Overrides the existing configuration.

kubectl apply -f manifest_file.yaml        

Create objects

kubectl create -f manifest_file.yaml        

Create objects in all manifest files in a directory

kubectl create -f ./dir        

Create objects from a URL

kubectl create -f ‘url’        

Delete an object

kubectl delete -f manifest_file.yaml?        

Namespaces

Shortcode = ns

Create namespace <name>

kubectl create namespace <namespace_name>        

List one or more namespaces

kubectl get namespace <namespace_name>        

Display the detailed state of one or more namespace

kubectl describe namespace <namespace_name>        

Delete a namespace

kubectl delete namespace <namespace_name>        

Edit and update the definition of a namespace

kubectl edit namespace <namespace_name>        

Display Resource (CPU/Memory/Storage) usage for a namespace

kubectl top namespace <namespace_name>        

Nodes

Shortcode = no

Update the taints on one or more nodes

kubectl taint node <node_name>        

List one or more nodes

kubectl get node        

Delete a node or multiple nodes

kubectl delete node <node_name>        

Display Resource usage (CPU/Memory/Storage) for nodes

kubectl top node        

Resource allocation per node

kubectl describe nodes | grep Allocated -A 5        

Pods running on a node

kubectl get pods -o wide | grep <node_name>        

Annotate a node

kubectl annotate node <node_name>        

Mark a node as unschedulable

kubectl cordon node <node_name>        

Mark node as schedulable

kubectl uncordon node <node_name>        

Drain a node in preparation for maintenance

kubectl drain node <node_name>        

Add or update the labels of one or more nodes

kubectl label node        

Pods

Shortcode = po

List one or more pods

kubectl get pod        

Delete a pod

kubectl delete pod <pod_name>        

Display the detailed state of a pods

kubectl describe pod <pod_name>        

Create a pod

kubectl create pod <pod_name>        

Execute a command against a container in a pod

kubectl exec <pod_name> -c <container_name> <command>        

Get interactive shell on a a single-container pod

kubectl exec -it <pod_name> /bin/sh        

Display Resource usage (CPU/Memory/Storage) for pods

kubectl top pod        

Add or update the annotations of a pod

kubectl annotate pod <pod_name> <annotation>        

Add or update the label of a pod

kubectl label pod <pod_name>        

Workloads Won’t Run:

kubectl get events --field-selector type=Warning --all-namespace

kubectl get nodes -o wide --label-columns topology.kubernetes.io/zone

kubectl resource-capacity --pods --util --sort cpu.util

kubectl get all --show-labels

kubectl lineage pod ${POD}

kail -l ${LABEL}s        

Debug Specific Pods

kubectl debug -it --image=debian $POD

kubectl label pod $POD app-

kubectl blame pod
        

The kubectl Logs Command

Tail Cluster Logs

tail -f /var/log/kube-apiserver.log        

Events

kubectl get events        

Tail Container Logs

Default Logs

kubectl logs podname -n namespace        

Specific Container Logs

kubectl logs podname -n namespace -c container_name        

All Containers

Kubectl logs podname -n namespace –all-containers=true        

Tail Logs from Pods

kubectl logs -l name=myLabel        

Limit the Number of Lines

kubectl logs podname -n namespace - -tail=100        

See the Live Logs or Follow

kubectl logs podname -n namespace -f        

Select Pods with Labels

kubectl logs -l app_name=nginx        

Timestamp Option-Based Log

kubectl logs —-timestamps=true -l app_name=nginx        

Multiple Pods

kubectl logs —-selector label_key=label_value        

All Container Logs

kubectl logs -l app_name=nginx –all-containers=true        

Previous Container Logs

kubectl logs podname -n namespace -p        

View Logs Since

kubectl logs podname -n namespace –since=1h        

Pod Events

kubectl describe pod podname        

Tail Logs from Nodes

Logs

/var/log/kubelet.log        

/var/log/kube-proxy.log        

The following command lets you view the CSI of CNI pod logs:

kubectl logs CNI_POD_NAME -n kube-system
kubectl logs CSI_POD_NAME -n kube-system        

Events

kubectl describe node node_name        

See Deployment Logs

kubectl describe deployment deployment_name
kubectl describe replicaset replicaset_name
kubectl describe pod
kubectl logs deployment/deployment_name        

Thanks For Reading!!!!