Mastering Pod Resource Management in Kubernetes

Pod resource management in Kubernetes is crucial for optimizing resource allocation and ensuring the stability and performance of your workloads. In this article, we’ll explore the key lessons of pod resource management through real-world examples and practical insights. Kubernetes provides a number of features that can be used to manage pod resources, including:

Resource requests and limits: You can specify the minimum and maximum amount of resources that a pod needs using resource requests and limits. This ensures that pods are allocated the resources they need to run, and that they do not consume more resources than they need.

Quality of Service (QoS) classes: Kubernetes provides three QoS classes: Guaranteed, Burstable, and Best Effort. You can assign a QoS class to a pod to specify the level of priority that it should receive when resources are allocated.

Horizontal Pod Autoscaling (HPA): HPA can automatically scale the number of replicas of a pod up or down based on the demand for the application. This helps to ensure that your application is always available and can handle spikes in traffic. (We will discuss HPA in our future article)

Understanding Resource Requests and Limits: In Kubernetes, you can specify two important resource parameters for pods: resource requests and resource limits.

Resource Requests: Think of resource requests as a pod’s polite way of asking for resources. It’s like a restaurant reservation. When you request resources, Kubernetes reserves them for your pod. Resource requests ensure that your pod gets a minimum amount of CPU and memory guaranteed to be available. If those resources aren’t available, your pod won’t be scheduled.

For example, you might have a web server pod that requests 0.5 CPU cores and 256MB of memory. Kubernetes will ensure these resources are available for the pod.

Resource Limits: Resource limits are like setting boundaries. They tell Kubernetes the maximum amount of CPU and memory a pod is allowed to use. Think of it as a buffet with limits on the number of plates you can fill. Setting limits prevents a single pod from consuming all available resources and affecting other pods.

For example, you might set a limit of 1 CPU core and 512MB of memory for a database pod.

Quality of Service (QoS)?classes in Kubernetes are a way to categorize and manage the resource allocation and prioritization of pods within a cluster. These classes help Kubernetes make decisions about how to distribute resources and prioritize workloads based on their resource requirements and behavior.

There are three QoS classes in Kubernetes:

Guaranteed?(QoS Class — Guaranteed): Pods in the Guaranteed QoS class have the highest priority for resource allocation. They are typically reserved for critical system components or applications that require a fixed amount of CPU and memory. Kubernetes ensures that these pods receive the requested resources without any oversubscription or resource contention.

Example: Cluster DNS pods, control plane components, or mission-critical applications with specific resource requirements.

Burstable?(QoS Class — Burstable): Pods in the Burstable QoS class have moderate priority. They are allowed to use resources beyond their resource requests up to their resource limits when available resources allow. Burstable pods can consume additional resources during periods of low cluster activity but may face resource constraints during high-demand periods. Kubernetes allows some degree of over commitment for CPU resources but not for memory.

Example: Most applications fall into this category, where they can burst to use more resources temporarily but are subject to resource limits.

BestEffort?(QoS Class — BestEffort): Pods in the BestEffort QoS class have the lowest priority. They have no resource requests or limits specified in their pod specifications. Kubernetes allocates leftover resources to BestEffort pods after satisfying the requests and limits of Guaranteed and Burstable pods. These pods are typically used for applications that can tolerate resource constraints and are not resource-critical.

Example: Testing or development pods where resource requirements are not well-defined.

Kubernetes uses QoS classes to make decisions regarding eviction and resource allocation. Here’s how these decisions are typically made:

Eviction Priority: When resources become scarce, Kubernetes may evict pods with lower QoS classes first to ensure higher-priority pods (e.g., Guaranteed) receive their requested resources.

Resource Allocation: Kubernetes allocates resources based on QoS classes. Guaranteed pods receive their full resource requests, Burstable pods receive resources within their limits, and BestEffort pods utilize leftover resources.

Resource Guarantees: Pods with higher QoS classes are guaranteed to receive their requested resources, ensuring their stability and performance.

Priority can be defined both manually and automatically.

Manually defining priority:

To manually define priority, you can use the spec.priority field in the pod specification. The?spec.priority?field is an integer value that can range from?-2147483648 to 2147483647.?Pods with a higher priority will be scheduled to nodes before pods with a lower priority.

Kubernetes automatically defining priority:

Kubernetes can also automatically define priority for pods. Kubernetes does this by considering the following factors:

QoS class: Pods with the Guaranteed QoS class have the highest priority, followed by pods with the Burstable QoS class, and then pods with the BestEffort QoS class. Pod priority: The pod priority can be set using the spec.priority field in the pod specification. Pods with a higher priority will be scheduled to nodes before pods with a lower priority. Resource usage: Kubernetes will also consider the resource usage of the pod when determining priority. Pods that are using more resources will be given a higher priority than pods that are using fewer resources.

The following table shows the priorities of pods based on their QoS class and priority value:

The following is example of pod specification file to illustrate our learnings from this article

# Pod with Guaranteed QoS, High Priority:
apiVersion: v1
kind: Pod
metadata:
  name: high-priority-guaranteed
spec:
  # Assign the "high-priority" priority class for this pod.
  priorityClassName: high-priority
  # Set a high priority value of 1000.
  priority: 1000
  containers:
    - name: high-priority-container
      image: nginx
      resources:
        # Specify resource requests for CPU and memory.
        requests:
          memory: "256Mi"
          cpu: "500m"
        # Define resource limits for CPU and memory.
        limits:
          memory: "1Gi"
          cpu: "1"

# Pod with Guaranteed QoS, Medium Priority:
apiVersion: v1
kind: Pod
metadata:
  name: medium-priority-guaranteed
spec:
  # Assign the "medium-priority" priority class for this pod.
  priorityClassName: medium-priority
  # Set a medium priority value of 500.
  priority: 500
  containers:
    - name: medium-priority-container
      image: nginx
      resources:
        # Specify resource requests for CPU and memory.
        requests:
          memory: "256Mi"
          cpu: "500m"
        # Define resource limits for CPU and memory.
        limits:
          memory: "1Gi"
          cpu: "1"

# Pod with Guaranteed QoS, Low Priority:
apiVersion: v1
kind: Pod
metadata:
  name: low-priority-guaranteed
spec:
  # Assign the "low-priority" priority class for this pod.
  priorityClassName: low-priority
  # Set a low priority value of 100.
  priority: 100
  containers:
    - name: low-priority-container
      image: nginx
      resources:
        # Specify resource requests for CPU and memory.
        requests:
          memory: "256Mi"
          cpu: "500m"
        # Define resource limits for CPU and memory.
        limits:
          memory: "1Gi"
          cpu: "1"

# Pod with Guaranteed QoS, Lowest Priority:
apiVersion: v1
kind: Pod
metadata:
  name: lowest-priority-guaranteed
spec:
  # Assign the "lowest-priority" priority class for this pod.
  priorityClassName: lowest-priority
  # Set the lowest priority value of 1.
  priority: 1
  containers:
    - name: lowest-priority-container
      image: nginx
      resources:
        # Specify resource requests for CPU and memory.
        requests:
          memory: "256Mi"
          cpu: "500m"
        # Define resource limits for CPU and memory.
        limits:
          memory: "1Gi"
          cpu: "1"
# The following sections are for Burstable and BestEffort combinations
---
apiVersion: v1
kind: Pod
metadata:
  name: high-priority-burstable
spec:
  priorityClassName: high-priority
  priority: 1000
  containers:
    - name: high-priority-container
      image: nginx
      resources:
        requests:
          memory: "256Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1"
---
apiVersion: v1
kind: Pod
metadata:
  name: medium-priority-burstable
spec:
  priorityClassName: medium-priority
  priority: 500
  containers:
    - name: medium-priority-container
      image: nginx
      resources:
        requests:
          memory: "256Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1"
---
apiVersion: v1
kind: Pod
metadata:
  name: low-priority-burstable
spec:
  priorityClassName: low-priority
  priority: 100
  containers:
    - name: low-priority-container
      image: nginx
      resources:
        requests:
          memory: "256Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1"
---
apiVersion: v1
kind: Pod
metadata:
  name: lowest-priority-burstable
spec:
  priorityClassName: lowest-priority
  priority: 1
  containers:
    - name: lowest-priority-container
      image: nginx
      resources:
        requests:
          memory: "256Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1"
---
apiVersion: v1
kind: Pod
metadata:
  name: medium-priority-besteffort
spec:
  priorityClassName: medium-priority
  priority: 500
  containers:
    - name: medium-priority-container
      image: nginx
---
apiVersion: v1
kind: Pod
metadata:
  name: low-priority-besteffort
spec:
  priorityClassName: low-priority
  priority: 100
  containers:
    - name: low-priority-container
      image: nginx
---
apiVersion: v1
kind: Pod
metadata:
  name: lowest-priority-besteffort
spec:
  priorityClassName: lowest-priority
  priority: 1
  containers:
    - name: lowest-priority-container
      image: nginx        

It is important to note that the priorities in the table are just a general guideline. Kubernetes may adjust the priority of a pod based on other factors, such as the resource usage of the pod and the state of the cluster.

You should choose the QOS class and priority for a pod based on the importance of the pod and the resources that it needs. For example, you may want to give a high priority to a pod that runs a critical application, such as a database or web server.

You should also be aware that the QOS class and priority of a pod can affect its performance. Pods with a higher priority are more likely to be scheduled to nodes and to have the resources they need to run. However, pods with a higher priority may also have a greater impact on the performance of other pods.

#Kubernetes #DevOps #ContainerOrchestration #LearningNuggets #K8sTips #GameOfKubes #K8s #platformengineering