Configuring Scheduler Profiles in Kubernetes

Kubernetes provides the ability to configure multiple scheduler profiles to control how pods get scheduled onto nodes. Scheduler profiles allow you to customize the scheduling behavior and policies for different types of workloads in your cluster.

In Kubernetes, a scheduler profile is defined as a configuration file in YAML format. This profile config specifies the various scheduling plugins to enable or disable, such as plugins for affinity/anti-affinity, resource spreading, node selectors, and more. By tuning these plugins in a profile, you can modify how the Kubernetes scheduler will prioritize and filter nodes when scheduling pods.

Once you have defined a custom scheduler profile configuration, there are two main ways to load it into the Kubernetes scheduler. One option is to specify the config file path directly in the kube-scheduler command when launching the scheduler. The other approach is to package the profile config into a ConfigMap resource, and reference that ConfigMap from the kube-scheduler deployment manifest.

With scheduler profiles configured, you can direct particular pods to use them by setting the "schedulerName" in the pod specifications. This allows pods to be scheduled using the customized scheduling behavior defined in that profile, instead of the default scheduler.

Overall, scheduler profiles provide a powerful way to tune scheduling policies tailored to the needs of various workloads in a Kubernetes cluster. Using multiple profiles allows you to optimize scheduling for specific pods or namespaces, beyond what the default scheduler can offer.

Creating a Scheduler Profile

To create a scheduler profile, you need to define a configuration file for it. This is a YAML file that specifies the plugins and plugin configurations for the profile.

Here is an example scheduler profile config:

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration

profiles:
  - schedulerName: my-custom-scheduler
    plugins:
      preFilter:
        enabled:
        - name: PodTopologySpread
      filter:
        enabled: 
        - name: NodeResourcesFit
        disabled:
        - name: NodePorts
      preScore:
        enabled:
        - name: InterPodAffinity
        disabled:
        - name: ImageLocality
      score:
        enabled:
        - name: NodeResourcesBalancedAllocation
        disabled:
        - name: NodeAffinity        

This defines a profile called "my-custom-scheduler" that enables certain plugins and disables others.

Some key things configured:

• preFilter: PodTopologySpread is enabled, which spreads pods across topology domains.
• filter: NodePorts is disabled so pod-to-node port requirements are ignored.
• preScore: InterPodAffinity is enabled to favor nodes with pods from same service.
• score: NodeResourcesBalancedAllocation is enabled to spread pods across nodes.

Using a Scheduler Profile

Once you have defined a scheduler profile config, you need to load it into the Kubernetes scheduler. There are two ways to do this:

1. Set the config file path in the scheduler command when launching the scheduler:

kube-scheduler --config=/path/to/custom-profiles.yaml        

2. Create a ConfigMap containing the profile config and specify that in the scheduler deployment manifest:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scheduler-profiles
  namespace: kube-system
data:
  custom-profiles.yaml: |
    # Contents of profile config file        

Then reference this in the scheduler deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: kube-scheduler
        args:
        - --configmap=scheduler-profiles/custom-profiles.yaml        

Once the custom profiles are loaded, you can direct pods to use them by setting the schedulerName in the pod spec.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  schedulerName: my-custom-scheduler        

This will make this pod use the "my-custom-scheduler" profile instead of the default scheduler.

Conclusion

Scheduler profiles in Kubernetes allow for customizable scheduling configurations for different types of workloads. By defining scheduler profiles in YAML config files, you can specify which plugins are enabled or disabled to tune the scheduling behavior. Loading these profiles into the Kubernetes scheduler can be done by setting the config file path directly or using a ConfigMap resource.

Once loaded, pods can be directed to use a specific scheduler profile by setting the "schedulerName" field in the pod spec. This causes those pods to be scheduled using the policies defined in that profile instead of the default scheduler.

The ability to configure multiple scheduler profiles provides a powerful way to optimize scheduling in Kubernetes clusters. Critical workloads can use profiles that prioritize resource spreading and availability. Batch workloads may leverage profiles customized for rapid focused scheduling. By pairing scheduler profiles with pods and namespaces, scheduling can be tailored at a very granular level.

Scheduler profiles give Kubernetes users more control over scheduling behavior than the default scheduler alone. They allow tailoring scheduling policies specifically to the needs of particular workloads or types of applications. This improves flexibility, optimization, and customization of Kubernetes scheduling overall.