Kubernetes Operator Explained

In recent years, Kubernetes has become the de facto standard for managing containerized applications at scale. With its rich set of APIs, Kubernetes handles the deployment, scaling, and operations of applications. However, as applications grow more complex—particularly those requiring intricate lifecycle management, like databases, message queues, or monitoring systems—standard Kubernetes resources like Deployments or StatefulSets often fall short. This is where Kubernetes Operators come in.

In this article, we will take a deep dive into Kubernetes Operators—what they are, how they work, and why they are useful. We’ll also cover how you can build a Kubernetes Operator using Go and compare this approach with traditional application deployments.

What is a Kubernetes Operator?

A Kubernetes Operator is an application-specific controller that extends Kubernetes' functionality by embedding domain-specific operational knowledge. Operators automate the full lifecycle of an application, using Kubernetes' native mechanisms and APIs. The core idea is to use the same declarative API used to manage standard resources like Pods, but for custom resources (CRDs) tailored to your application's needs.

The Operator pattern originated at CoreOS as a solution to automate complex applications on Kubernetes clusters, including managing Kubernetes itself and the etcd key-value store. Work on Operators continued through an acquisition by Red Hat, leading to the 2018 release of the open-source Operator Framework and SDK

At its core, an Operator does the following:

• Defines Custom Resources (CRDs): CRDs extend Kubernetes to recognize and manage new resource types specific to your application.
• Automates Lifecycle Management: The Operator constantly monitors the application’s desired state and reconciles it with the actual state, managing complex tasks like updates, scaling, and failovers.
• Handles Advanced Automation: Operators can perform advanced, domain-specific operations, like database migrations or partition rebalancing for distributed systems like Kafka.

How Does a Kubernetes Operator Work?

A Kubernetes cluster is a collection of nodes (computers), each of which can run tasks. Within this cluster, the basic unit of work and replication is the pod—a group of one or more Linux containers that share resources like networking, storage, and memory.

At a high level, a Kubernetes cluster is divided into two planes.

• Control Plane: This plane, in essence, is Kubernetes itself. It orchestrates the cluster and implements Kubernetes’ API. The control plane comprises multiple pods to handle tasks like scheduling, management, and control loops.
• Application (Data) Plane: This is where application workloads run. It includes nodes dedicated to application pods, while certain nodes may be allocated specifically for control plane components, providing redundancy for critical services.

The controllers of the control plane implement control loops that repeatedly compare the desired state of the cluster to its actual state. When the two diverge, a controller takes action to make them match. Operators extend this capability, managing complex application lifecycle tasks using the same pattern.

The diagram below shows the main control plane components alongside three worker nodes running application workloads.:

Kubernetes Operators rely on two core components:

1. Custom Resource Definitions (CRDs)

A Custom Resource Definition (CRD) is the schema used to define a new resource type that extends Kubernetes’ built-in resources. CRDs allow you to represent your application’s state and configuration as custom resources. For example, if you're managing a database, you could create a custom resource named MyDatabase which specifies the size, backup schedules, replicas, or other configuration details unique to the database instance.

In this context:

• Custom resources (CRs) represent the desired state of the application.
• They enable you to declare application-specific configurations using Kubernetes manifests, much like you do with built-in resources like Pods or Services.

2. Custom Controller

The Custom Controller is the operational logic that actively monitors the custom resources defined by your CRD. It continuously compares the current state of the system with the desired state defined in the CRD, and takes corrective actions if there is a deviation. The controller interacts with underlying Kubernetes resources (such as Pods, StatefulSets, or ConfigMaps) to manage the lifecycle of the application.

The reconciliation process typically follows these steps:

• Monitor the custom resource: The controller watches for changes to the CR (e.g., MyDatabase) using the Kubernetes API.
• Compare states: The controller compares the actual state of the resources with the desired state defined in the custom resource.
• Take action: If the actual state doesn't match the desired state (e.g., fewer replicas running than specified), the controller takes action to reconcile the difference (e.g., by creating or deleting Pods).

For instance, if the MyDatabase resource specifies that there should be three replicas of the database, but only two are running, the controller will create another pod to meet the specified number of replicas.

The Reconciliation Loop

Operators use the reconciliation loop pattern, which is a continuous process to ensure that the current state of the system aligns with the desired state defined in the custom resource. The reconciliation loop runs continuously, automatically detecting changes in the system (e.g., pod crashes or configuration updates) and making adjustments to bring the system back into compliance.

This ensures the application remains in a consistent state, with minimal manual intervention. The loop is central to how Operators deliver automated management of even complex, stateful applications.

Benefits of Deploying an Application as an Operator

1. Advanced Automation: Operators can automate complex lifecycle operations (e.g., backups, upgrades, failovers) that require domain-specific knowledge. This level of automation is hard to achieve with just Kubernetes' core resources like Deployments and StatefulSets.
2. Better Lifecycle Management: Operators handle tasks like monitoring the application’s health, performing self-healing, and handling automatic scaling. They allow you to move beyond basic lifecycle management and implement sophisticated operational logic.
3. Encapsulation of Domain Knowledge: Operators embed domain-specific logic, allowing them to handle more complex operations tailored to the application. For example, a Kafka Operator might know how to reassign partitions when brokers are scaled up or down.
4. Self-Healing Capabilities: Operators detect failures and take corrective actions, ensuring high availability and minimizing downtime. If an application goes down or misbehaves, the Operator restores it to the desired state without human intervention.
5. Declarative Management: Operators use declarative APIs, so you define the desired state, and the Operator continuously reconciles it with the actual state.
6. Scaling Beyond Deployments: For simple applications, you can use a Deployment or StatefulSet. But for more complex applications requiring customized scaling, specialized rolling updates, or maintenance tasks, an Operator is a better choice. For example, if scaling a database requires rebalancing shards or partitions, an Operator can handle this in a domain-specific way.

Building/Deploying a Kubernetes Operator in Go

Go is a popular choice for building Kubernetes Operators, thanks to its strong support for Kubernetes client libraries like client-go. With client-go, you can directly interact with the Kubernetes API, making it easier to develop complex, application-specific logic within your Operator.

• Set up CRDs and controllers: First, define the custom resource types your Operator will manage.
• Implement the reconciliation loop: This loop will monitor and ensure that the application's state is continuously reconciled with the desired configuration.
• Test the Operator: Use Kubernetes tools like minikube or kind to test it locally.

For a more in-depth understanding of client-go, see my article "Overview of Kubernetes Client Library"

When building an Operator, the Operator-SDK is a common tool to help scaffold and manage the operator's logic. Other Operator tools include Kopf (Python-based), kubebuilder (a Go framework), Ansible , and Helm —choose based on your preferred programming language and the complexity of your Operator.

Here’s an approach to building a Kubernetes Operator in Go using the Operator SDK.

Steps to Build a Kubernetes Operator in Go

1. Set up the Development Environment: Install Go and the Operator SDK.
2. Initialize the Project:

mkdir -p $HOME/projects/memcached-operator
cd $HOME/projects/memcached-operator
# we'll use a domain of example.com
# so all API groups will be <group>.example.com
operator-sdk init --domain example.com --repo github.com/example/memcached-operator        

3. Define Your Custom Resource (CRD): Create a new CRD and its associated controller.

$ operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller
Writing scaffold for you to edit...
api/v1alpha1/memcached_types.go
controllers/memcached_controller.go
...        

4. Write Reconciliation Logic:

• Implement the business logic for managing your custom resource inside the generated controller.
• The reconciliation loop ensures that the actual state of the application matches the desired state defined in the custom resource.

Example of a Reconciliation Loop in Go:

import (
	ctrl "sigs.k8s.io/controller-runtime"

	cachev1alpha1 "github.com/example/memcached-operator/api/v1alpha1"
	...
)

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  // Lookup the Memcached instance for this reconcile request
  memcached := &cachev1alpha1.Memcached{}
  err := r.Get(ctx, req.NamespacedName, memcached)
  ...
}        

5. Test and Deploy the Operator:

• Use Kubernetes minikube or kind to test the Operator locally
• Once tested locally, build and push the Operator image to a registry:

make docker-build docker-push
        

• Deploy the Operator to your Kubernetes cluster:

make deploy IMG=<your-image>        

6. Manage Custom Resources

Once the Operator is running, create instances of your custom resource:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3
  containerPort: 11211        

Create the CR:

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml        

Ensure that the memcached operator creates the deployment for the sample CR with the correct size:

$ kubectl get deployment
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
memcached-sample                        3/3     3            3           1m        

Check the pods and CR status to confirm the status is updated with the memcached pod names:

$ kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
memcached-sample-6fd7c98d8-7dqdr      1/1       Running   0          1m
memcached-sample-6fd7c98d8-g5k7v      1/1       Running   0          1m
memcached-sample-6fd7c98d8-m7vn7      1/1       Running   0          1m
        

$ kubectl get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  clusterName: ""
  creationTimestamp: 2018-03-31T22:51:08Z
  generation: 0
  name: memcached-sample
  namespace: default
  resourceVersion: "245453"
  selfLink: /apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/memcached-sample
  uid: 0026cc97-3536-11e8-bd83-0800274106a1
spec:
  size: 3
status:
  nodes:
  - memcached-sample-6fd7c98d8-7dqdr
  - memcached-sample-6fd7c98d8-g5k7v
  - memcached-sample-6fd7c98d8-m7vn7
        

The Operator will monitor these resources and manage them according to the logic you've defined.

7. Monitor and Update: Ensure that the Operator is continually managing the application's lifecycle by monitoring and updating as necessary.

More details, please refer to go operator tutorial

Other Operator Tools

Other open-source tools available for building Operators include Kopf for Python, Kubebuilder from the Kubernetes project, and the Java Operator SDK .

Conclusion

A Kubernetes Operator enables advanced automation of application lifecycle management by embedding domain-specific knowledge within Kubernetes controllers. Instead of managing applications with standard Kubernetes resources like Deployments or StatefulSets, Operators allow you to manage complex applications with custom logic, automating tasks like scaling, backups, and upgrades.

Building an Operator in Go using tools like Operator SDK allows you to easily extend Kubernetes' capabilities and integrate custom logic into the platform. Deploying applications as Operators provides significant benefits, especially for complex stateful applications that need sophisticated management beyond what standard Kubernetes resources can provide. Kubernetes Operators

Operators help move towards fully autonomous applications that self-manage, reduce manual intervention, and improve reliability in production environments.

References

To deepen your understanding of Kubernetes Operators and containerized environments, check out the following resources:

• "Programming Kubernetes" by Michael Hausenblas and Stefan Schimanski - This book explores Kubernetes' API and architecture, and includes a comprehensive guide on writing Kubernetes Operators.
• "Kubernetes Operators" by Jason Dobies and Joshua Wood - This book provides a practical introduction to Operators, including how to build and manage them.
• Best practices for building Kubernetes Operators and stateful apps