Node Maintenance Commands In Kubernetes

In a Kubernetes cluster, there may be situations where you need to perform maintenance tasks on a node, such as kernel upgrades, hardware maintenance, or application updates. To handle these scenarios, Kubernetes provides utility commands called kubectl drain, kubectl cordon, and kubectl uncordon. These commands help you manage nodes in your cluster and ensure that your applications remain available during maintenance windows by safely rescheduling pods on other nodes.

The kubectl drain command is used to safely evict all the pods from a node before performing maintenance operations. It marks the node as unschedulable to prevent new pods from being scheduled on it, evicts the current pods from the node, and ensures that the evicted pods are recreated on other nodes in the cluster. This command can be customized with various options to handle specific scenarios, such as ignoring daemonsets, deleting local data, or forcibly evicting pods.

Before draining a node, it's often recommended to first mark it as unschedulable using the kubectl cordon command. This command ensures that no new pods are scheduled on the node, preparing it for the subsequent draining operation. Once the maintenance tasks are completed and the node is ready to receive workloads again, the kubectl uncordon command can be used to mark the node as schedulable.

These commands work together to provide a smooth and controlled process for managing nodes in a Kubernetes cluster. By following a typical workflow of cordoning, draining, performing maintenance, and uncordoning, you can ensure minimal disruption to your applications and maintain high availability during maintenance periods.

kubectl drain

The kubectl drain command safely evicts all the pods from a node before you perform any maintenance operation on it. This command does the following:

1. Marks the node as unschedulable to prevent new pods from being scheduled on the node.
2. Evicts the current pods from the node.
3. Ensures that the evicted pods are recreated on other nodes in the cluster.

Here's the syntax:

kubectl drain <node-name> [--ignore-daemonsets] [--delete-local-data] [--force] [--grace-period=-1]        

• <node-name>: The name of the node to be drained.
• --ignore-daemonsets: Skip draining daemonset pods from the node.
• --delete-local-data: Continues draining and deleting pods even if they are using emptyDir volumes.
• --force: Forcibly evict pods even if they are still running
• --grace-period=-1: Forcibly terminate the pod immediately without respecting the specified grace period.

Example:

kubectl drain node01 --ignore-daemonsets        

This command will drain all the pods from node01, except the daemonset pods.

kubectl cordon

The kubectl cordon command marks a node as unschedulable, which means that no new pods will be scheduled on that node. This command is typically used before performing maintenance tasks on a node or as a preparatory step before draining the node.

Here's the syntax:

kubectl cordon <node-name>        

• <node-name>: The name of the node to be cordoned.

Example:

kubectl cordon node01        

This command will mark node01 as unschedulable.

kubectl uncordon

The kubectl uncordon command marks a node as schedulable again after maintenance tasks have been completed or after the node has been cordoned or drained.

Here's the syntax:

kubectl uncordon <node-name>        

• <node-name>: The name of the node to be uncordoned.

Example:

kubectl uncordon node01        

This command will mark node01 as schedulable again.

Typical Workflow

A typical workflow for performing maintenance tasks on a node would be:

1. Run kubectl cordon <node-name> to mark the node as unschedulable.
2. Run kubectl drain <node-name> to evict all the pods from the node.
3. Perform your maintenance tasks on the node.
4. Run kubectl uncordon <node-name> to mark the node as schedulable again.

After running kubectl uncordon, new pods can be scheduled on the node, and the pods that were evicted during the drain operation will be rescheduled on the available nodes in the cluster.

Note that kubectl drain is a safe operation that ensures that your applications remain available during the maintenance window by rescheduling the pods on other nodes. However, it's always a good practice to monitor the cluster and the application health during and after the maintenance tasks.

Conclusion

The kubectl drain, kubectl cordon, and kubectl uncordon commands are essential tools in the Kubernetes ecosystem for managing nodes and ensuring minimal disruption to applications during maintenance periods. By following the typical workflow of cordoning a node to prevent new pod scheduling, draining the node to safely evict existing pods, performing the required maintenance tasks, and finally uncordoning the node to make it schedulable again, you can maintain high availability for your applications while carrying out necessary maintenance operations.

It's important to note that while the kubectl drain command aims to safely evict pods, it's always a good practice to monitor the cluster and application health during and after the maintenance tasks. Additionally, customizing the drain command with appropriate options, such as ignoring daemonsets or forcibly terminating pods, can help tailor the process to your specific requirements.

By understanding and utilizing these Kubernetes commands effectively, you can streamline your cluster management processes, minimize downtime, and ensure a smooth and controlled maintenance experience for your applications running on Kubernetes.