Controllers and Operators

This post is longer than usual and covers Controllers and Operators, these two are closely related but are not the same.

Controllers are a core part of Kubernetes, designed to manage and maintain the desired state of the cluster by observing and reacting to changes. They are essentially control loops that continuously work to reconcile the current state of resources with the desired state. Here’s a breakdown of their key aspects:

Core Functions

• Reconciliation Loop: Controllers operate using a reconciliation loop. This involves observing the current state of the cluster, comparing it to the desired state, and taking actions to align the actual state with the desired state.
• Monitoring: Controllers watch the API server for changes to resources they are responsible for. They respond to events, such as the creation, deletion, or modification of resources.
• State Management: They continuously strive to match the observed state of the cluster with the desired state. If a discrepancy is detected, the controller takes corrective actions.
• Automation: Controllers automate tasks, such as scaling applications and rolling out updates. They manage these processes without requiring manual intervention.

Types of Controllers

• Built-in Controllers: Kubernetes includes built-in controllers that manage core resources, such as ReplicaSets, Deployments, and StatefulSets.
• Custom Controllers: Kubernetes allows you to create custom controllers to manage specific application or domain logic. Custom controllers can extend Kubernetes functionality?and automate complex application lifecycle management.
• Deployment Controller: Manages ReplicaSets to control the rollout of application updates. When an update occurs, the Deployment controller creates a new ReplicaSet and gradually shifts traffic to it. It also handles rollbacks.
• ReplicaSet Controller: This controller ensures that a specified number of pod replicas are running at any time. It works to maintain the desired number of pod instances, creating or deleting pods to match the configuration.
• StatefulSet Controller: Manages stateful applications, maintaining the identity and order of pods. It ensures that pods are created and deleted in a controlled, ordered way.
• DaemonSet Controller: Ensures that a copy of a pod runs on each node (or a subset of nodes) in the cluster. Useful for node-level services and maintenance tasks.
• Operator: A more advanced type of controller that manages custom resources (CRDs), allowing you to encapsulate operational knowledge for a specific application in an automated way, operators extend the Kubernetes API.
• Horizontal Pod Autoscaler (HPA): Automatically scales the number of pods in a deployment or replica set based on observed CPU utilization or other metrics.
• Ingress Controller: Manages Ingress resources, which define how external traffic is routed to services within the cluster.
• Pod Controller: Used for orchestrating applications, such as DaemonSets and StatefulSets.

How do they work?

• Declarative Configuration: Controllers use declarative configurations where the desired state is specified, and the controller is responsible for achieving that state.
• Specialization: Each controller is highly specialized, focusing on its specific area of responsibility within the cluster.
• Self-Healing: Controllers contribute to the self-healing capabilities of Kubernetes by automatically replacing failed pods and ensuring the desired number of replicas are running.
• Watch API: Controllers use the Kubernetes watch API to receive a stream of updates when resources change.
• Custom Resource Definitions (CRDs): Operators, a type of controller, often utilize CRDs to extend Kubernetes and introduce new domain-specific resources.

Implementation

• Controllers can be implemented using various programming languages, though many are written in Go due to the client libraries available.
• They commonly run as background processes, continuously monitoring the state of the cluster.
• Controllers avoid concurrency issues through optimistic locking.
• Controllers are designed to be idempotent, meaning that multiple calls to reconcile an unchanged resource must produce the same effect each time.

Benefits

• Automation: Controllers automate many of the operational tasks associated with managing applications in Kubernetes.
• Simplified Operations: Controllers abstract away the complexity of managing individual pods and allow users to focus on defining the desired state of their applications.
• Improved Reliability: Self-healing and scaling capabilities ensure that applications are highly available and can handle fluctuating loads.
• Extensibility: Custom controllers and operators allow you to extend Kubernetes to meet specific application needs.

Operators are a method to extend Kubernetes to automate the management of the entire lifecycle of a particular application. They are custom controllers that use Custom Resource Definitions (CRDs) to encapsulate operational knowledge for a specific application in an algorithmic and automated form. Operators build on Kubernetes abstractions to automate the entire lifecycle of the software they manage. They also serve as a packaging mechanism for distributing applications on Kubernetes, and they monitor, maintain, recover, and upgrade the software they deploy.

• Custom Resource Definitions (CRDs): CRDs allow users to create custom Kubernetes objects, to store any data they wish. CRDs extend the Kubernetes API by adding custom resources to your Kubernetes cluster, which are then used as if they were native resources. A CRD defines a CR; it’s analogous to a schema for the CR data.
• Controllers: Operators are a type of controller. Controllers are part of the Kubernetes control plane, and they monitor and act on standard Kubernetes resources, enhancing platform behavior and adding new platform features.
• Custom Controllers: An operator is a custom Kubernetes controller watching a CR type and taking application-specific actions to make reality match the spec in that resource.
• Reconciliation: Operators work by extending the Kubernetes control plane and API. In its simplest form, an Operator adds an endpoint to the Kubernetes API, called a custom resource (CR), along with a control plane component that monitors and maintains resources of the new type. The Operator then takes action based on the resource’s state.
• Automation: Operators encode in software the skills of an expert administrator, automating tasks that usually require human intervention. They automate the lifecycle of the software they manage.
• Self-Healing: Operators can perform more advanced health detection and healing than can be achieved via Kubernetes’s generic self-healing. They can monitor, maintain, recover, and upgrade the software they deploy.
• Extensibility: Operators extend the Kubernetes API, providing application-specific automation in terms familiar to a large and growing community.

Operators are also a way to package, run, and maintain a Kubernetes application. A Kubernetes application is designed to use and to operate in concert with Kubernetes facilities and tools. They extend the Kubernetes control plane and API.

Operators can be used to manage various kinds of applications, such as:

• Stateful Applications: Operators are particularly useful for managing stateful applications with complex requirements like storage, networking, and identity.
• Databases: Operators can manage clusters of database servers, handling tasks like configuration, installation, backups, recovery, and upgrades. For example, the etcd operator manages the etcd key-value store. Other popular operators include those for PostgreSQL, MongoDB, and Redis.
• Other Complex Applications: Operators can also manage other complex applications, automating tasks such as scaling, upgrades, and configuration.

How Controllers and Operators are different?

Controllers and operators are both key components in Kubernetes for managing applications, but they differ in their scope and complexity.

Controllers

• Function: A controller actively monitors and maintains a set of Kubernetes resources to ensure they match the desired state They run in a continuous loop, observing the current state of resources and making changes to reach the target state
• Scope: Controllers typically manage standard Kubernetes resources such as ReplicaSets, Deployments, and Services.
• Complexity: Controllers generally implement simpler reconciliation processes.
• Customization: Controllers can be customized to watch and act upon specific fields in a resource definition, such as metadata and ConfigMaps, which can be used to store target state definitions. However, Custom Resource Definitions (CRDs) are recommended over plain ConfigMaps for custom target state specification.
• Examples: Examples of controllers include the Deployment controller, the StatefulSet controller, and the ReplicaSet controller. Kubernetes itself has built-in controllers that manage its standard resources.
• Purpose: Controllers enhance platform behavior and add new platform features. They are a way to extend how Kubernetes manages its resources.

Operators

• Function: An operator is a type of controller that extends Kubernetes by using Custom Resource Definitions (CRDs) to encapsulate operational knowledge for a specific application in an automated form.
• Scope: Operators interact with CRDs and manage the full lifecycle of an application, including installation, scaling, backups, recovery, and upgrades.
• Complexity: Operators are considered a more sophisticated reconciliation process compared to controllers. They encapsulate complex application domain logic Operators manage applications with bespoke requirements.
• Customization: Operators can perform application-specific actions, such as scaling a complex app, upgrading application versions, or managing specialized hardware.
• CRDs: Operators use CRDs to define custom resources, which provide a declarative API for managing specific applications and storing their state CRDs allowing for the creation of new concepts within the Kubernetes platform.
• Purpose: Operators are designed to automate tasks that usually require a human operator with expertise in both Kubernetes and the application being managed. They extend Kubernetes to automate the management of the entire lifecycle of a particular application, treating the application as a first-class citizen of Kubernetes. Operators aim to reduce operational effort and cost and increase service reliability.
• Analogy: Operators can be thought of as software Site Reliability Engineers (SREs) that encode the skills of expert administrators.

Key Differences Summarized

• Scope of Management: Controllers manage standard Kubernetes resources, while operators manage applications using custom resources defined by CRDs.
• Complexity: Operators are generally more complex and application-specific than controllers.
• Customization: Operators provide a high level of customization and can perform complex application-specific tasks, while controllers primarily manage the lifecycle of Kubernetes resources.
• CRDs: Operators utilize CRDs as a central part of their implementation to provide a declarative API for managing an application, whereas controllers typically work with built-in Kubernetes resources and may use ConfigMaps to store related data.

In essence, a controller is a general mechanism for managing resources within Kubernetes, while an operator is a specialized controller that extends Kubernetes by using CRDs to manage a specific application. Operators are also considered to be a more sophisticated evolutionary step from controllers.