Kubernetes - Taints, Tolerations, Node Selectors & Node Affinity

Taints, Tolerations, Node Selectors & Node Affinity are mechanisms for pod scheduling onto nodes.

Before we start lets create the kind cluster: Refer this article for installation -

https://www.dhirubhai.net/pulse/kubernetes-install-kind-create-multi-node-cluster-kiran-kulkarni-2ya8c

1. Node Selector:

Pods run on nodes with specific labels

Step 1: Add the label to the node

kubectl label nodes simple-multinode-cluster-worker2 prod=true        

Step 2: Specify the same label in the nodeSelector field of the pod

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
  nodeSelector:
    prod: "true"        

Note: Check the Node field you can see the pod is running on simple-multinode-cluster-worker2

2. Taints and Tolerations:

• Nodes : Taints repel pods from nodes that do not match the toleration
• Pods: Tolerations are added to pods to schedule onto the tainted nodes.

Effects:

• NoSchedule: Pods without toleration are not scheduled.
• PreferNoSchedule: It prefers not to schedule the pods. If no other nodes are available then it will schedule the pod because of the resource contraint.
• NoExecute: If the Pods are already running on the node they are evicted without toleration.

Step 1: Add a taint to the node

kubectl taint nodes simple-multinode-cluster-worker prod=true:NoSchedule        

Step 2: Add the toleration to the pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
  tolerations:
    - key: prod
      operator: Equal
      value: true
      effect: NoSchedule        

To Understand how it works. Let us create only one node and create 2 scenarios:

1. Tainted node and Pod without Toleration.

kubectl taint nodes simple-multinode-cluster-worker prod=true:NoSchedule        

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx        

Note: This pod will not run on the node as it does not have the toleration.

2. Tainted node and Pod with Toleration.

kubectl taint nodes simple-multinode-cluster-worker prod=true:NoSchedule        

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
  tolerations:
    - key: prod
      operator: Equal
      value: true
      effect: NoSchedule        

Step 3: Combine Node Selectors and Tolerations:

Is it necessary to combine both? The answer is yes in certain conditions.

Combining Node Selectors with Taints and Tolerations adds two layers of control to pod scheduling. This ensures only specific pods can run on specific nodes.

kubectl label nodes simple-multinode-cluster-worker environment=prod
kubectl taint nodes simple-multinode-cluster-worker environment=prod:NoSchedule         

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
  nodeSelector:
    prod: true 
  tolerations:
    - key: prod
      operator: Equal
      value: true
      effect: NoSchedule         

Use Cases:

• Node Selector: Use it for Simple and Straightforward placement of pods.
• Taints/Tolerations: Dedicate nodes for specific teams or workloads
• Combined: Special nodes reserved for very specific pods (e.g., production-only nodes)

3. Node Affinity

Node Affinity is a more advanced and flexible alternative to nodeSelector for controlling pod scheduling.

It allows you to define rules to attract pods to specific nodes based on node labels, using logical operators like In, NotIn, Exists, etc.

Types of Node Affinity

1. requiredDuringSchedulingIgnoredDuringExecution (Hard Rule):

• Pods must be scheduled on nodes that meets the criteria.
• Pods will be pending if the nodes are not matched.

2. preferredDuringSchedulingIgnoredDuringExecution (Soft Rule):

• Try to schedule pods on nodes that meet the criteria.
• If no node matches, the pods will be scheduled else where.

Example: Scheduling Pods with Node Affinity

You want a pod to run on:

• Must: A node labeled environment=prod.
• Preferably: A node in region us-east-1.

kubectl label nodes simple-multinode-cluster-worker environment=prod zone=us-east-1 

kubectl label nodes simple-multinode-cluster-worker2 environment=prod zone=us-west-1 
1  

kubectl label nodes simple-multinode-cluster-worker3 environment=dev zone=us-east-1         

apiVersion: v1
kind: Pod
metadata:
  name: nginx-affinity
spec:
  containers:
    - name: nginx
      image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: environment
                operator: In
                values: [prod]
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1 
          preference:
            matchExpressions:
              - key: region
                operator: In
                values: [us-east-1]        

Note: Check the Node field you can see the pod is running on simple-multinode-cluster-worker

• The pod must schedule on node1 or node2 (both have environment=prod).
• prefers node1(simple-multinode-cluster-worker) since it’s in us-east-1 but will use node2 if node1 is unavailable or busy.
• node3 is ignored (fails environment=prod).