Build Custom Kubernetes Operators with Ease Using TypeScript

Kubernetes operators offer powerful ways to automate cluster management. While tools like the Operator SDK and Kubebuilder simplify development (often using Go), this article demonstrates how to build a basic Kubernetes operator using TypeScript. We'll create a simple operator that deploys resources based on a Custom Resource Definition (CRD), providing a foundation for understanding TypeScript's role in Kubernetes automation.

Background

If you're interested in exploring alternatives to Go-based operator development, you may find these past articles helpful:

• Cloud Native Applications with Kubebuilder and Kind: Kubernetes Operators
• Testing the Operator SDK and Making a Prefetch Mechanism for Kubernetes

Prerequisites

To follow along, you'll need:

• Kind (for creating a local Kubernetes cluster)
• Docker
• Kustomize
• Node.js
• TypeScript

Project Resources

The source code for this example is available on GitHub. You can also find a pre-built Docker image. For further inspiration, check out Nodeshift's Operator in JavaScript.

Operator-SDK (Go) vs. TypeScript Operator

When choosing between these approaches, consider these key differences:

Operator-SDK (Go)

• Automatic generation of RBAC rules, boilerplate code, etc.
• Built-in tooling for building, deployment, and management.
• Requires knowledge of Go.

TypeScript Operator

• Requires manual handling of these configurations.
• Relies on external tools (often integrated into a CI system).
• Leverages familiarity with TypeScript/JavaScript.

Key Takeaways:

• Operator-SDK (Go): Provides greater structure and out-of-the-box convenience, potentially streamlining development.

• Custom TypeScript Operator: Offers flexibility and potential for wider developer accessibility, especially for JavaScript/TypeScript teams.

Example Implementation (TypeScript): In this guide, we used GitHub Actions to automate the build and deployment process. Although there's more manual setup, you gain fine-grained control over resources and the development process.

Let's get started!

For testing and development, we'll use Kind (Kubernetes in Docker) to quickly set up a local Kubernetes cluster. Kind's lightweight nature makes it ideal for this purpose. Here's how to create your cluster:

? kind create cluster
Creating cluster "kind" ...
 ? Ensuring node image (kindest/node:v1.21.1) ??
 ? Preparing nodes ??
 ? Writing configuration ??
 ? Starting control-plane ???
 ? Installing CNI ??
 ? Installing StorageClass ??
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community ??        

Before diving into the specifics, let's first see the operator in action. Rest assured, I'll explain the creation of these files shortly!

Configuring Your Operator:

Your operator relies on specific permissions and components within the Kubernetes cluster. We'll use Kustomize to streamline the creation of these resources:

? kustomize build resources/ | kubectl apply -f -
namespace/ts-operator created
customresourcedefinition.apiextensions.k8s.io/mycustomresources.custom.example.com created
serviceaccount/ts-operator created
clusterrole.rbac.authorization.k8s.io/mycustomresource-editor-role created
clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created
deployment.apps/ts-operator created

? kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
kube-system          coredns-558bd4d5db-284q5                     1/1     Running             0          21m
kube-system          coredns-558bd4d5db-5qs64                     1/1     Running             0          21m
kube-system          etcd-kind-control-plane                      1/1     Running             0          21m
kube-system          kindnet-njtns                                1/1     Running             0          21m
kube-system          kube-apiserver-kind-control-plane            1/1     Running             0          21m
kube-system          kube-controller-manager-kind-control-plane   1/1     Running             0          21m
kube-system          kube-proxy-d2gkx                             1/1     Running             0          21m
kube-system          kube-scheduler-kind-control-plane            1/1     Running             0          21m
local-path-storage   local-path-provisioner-547f784dff-tp6cq      1/1     Running             0          21m
ts-operator          ts-operator-86dbcd9f9c-xwgdt                 0/1     ContainerCreating   0          23s        

Interacting with the Operator:

Now, let's put our operator to the test. To interact with it, we'll create a sample instance of our Custom Resource Definition (CRD):

? kubectl apply -f resources/mycustomresource-sample.yaml
mycustomresource.custom.example.com/mycustomresource-sample created

? kubectl apply -f resources/mycustomresource-sample.yaml
mycustomresource.custom.example.com/mycustomresource-sample configured

? kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
kube-system          coredns-558bd4d5db-284q5                     1/1     Running             0          8h
kube-system          coredns-558bd4d5db-5qs64                     1/1     Running             0          8h
kube-system          etcd-kind-control-plane                      1/1     Running             0          8h
kube-system          kindnet-njtns                                1/1     Running             0          8h
kube-system          kube-apiserver-kind-control-plane            1/1     Running             0          8h
kube-system          kube-controller-manager-kind-control-plane   1/1     Running             0          8h
kube-system          kube-proxy-d2gkx                             1/1     Running             0          8h
kube-system          kube-scheduler-kind-control-plane            1/1     Running             0          8h
local-path-storage   local-path-provisioner-547f784dff-tp6cq      1/1     Running             0          8h
ts-operator          ts-operator-86dbcd9f9c-xwgdt                 1/1     Running             0          8h
workers              mycustomresource-sample-644c6fdf78-75hh7     1/1     Running             0          2m9s
workers              mycustomresource-sample-644c6fdf78-fv5n8     1/1     Running             0          2m9s
workers              mycustomresource-sample-644c6fdf78-hprt7     0/1     ContainerCreating   0          1s

? kubectl delete -f resources/mycustomresource-sample.yaml
mycustomresource.custom.example.com "mycustomresource-sample" deleted        

Understanding Operator Activity: Examining Logs

Your operator generates logs that offer valuable insights into its behavior. Let's look at logs that correspond to creating, updating, and deleting a Custom Resource:

? node_modules/ts-node/dist/bin.js src/index.ts
7/22/2021, 8:51:54 PM: Watching API
7/22/2021, 8:51:54 PM: Received event in phase ADDED.
7/22/2021, 8:52:04 PM: Received event in phase MODIFIED.
7/22/2021, 8:53:39 PM: Received event in phase ADDED.
7/22/2021, 8:53:40 PM: Nothing to update...Skipping...
7/22/2021, 8:53:40 PM: Received event in phase MODIFIED.
7/22/2021, 8:56:15 PM: Received event in phase ADDED.
7/22/2021, 8:56:20 PM: Received event in phase DELETED.
7/22/2021, 8:56:20 PM: Deleted mycustomresource-sample        

Automated Image Building and Pushing

One significant advantage of this approach is streamlined image management. We'll use GitHub Actions to automatically build and push your operator's Docker image to their free container registry. This process is transparent and requires no additional configuration on your repository.

• Branch-Based Management: Actions conveniently handles image creation and tagging, matching the relevant branch name.
• Efficiency: This integration saves you from manual build and push steps.

You can view the results here.

name: Create and publish a Docker image

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

  workflow_dispatch:

jobs:
  build-and-push-image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Log in to the Container registry
        uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      - name: Build and push Docker image
        uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}        

Local Development and Debugging

For streamlined development and testing, you can easily run your operator locally using ts-node. Here's how:

? node_modules/ts-node/dist/bin.js src/index.ts
7/22/2021, 8:51:54 PM: Watching API
7/22/2021, 8:51:54 PM: Received event in phase ADDED.
7/22/2021, 8:52:04 PM: Received event in phase MODIFIED.
7/22/2021, 8:52:10 PM: Received event in phase DELETED.
....        

Why ts-node?

• Zero Configuration: Assuming ts-node is a development dependency, you avoid extra setup for local execution.
• Rapid Iteration: Test code changes without rebuilding the Docker image each time.

Using the Docker Image

While less convenient for rapid iteration, the Docker image could be used locally with the right configuration. This might be beneficial for testing the containerized behavior of your operator.

Now let’s see the code

Enough words, let’s see code, I have added comments and explanations where it seemed necessary, but if you have any question feel free to ask.

/* eslint-disable @typescript-eslint/no-explicit-any */
/* eslint-disable @typescript-eslint/no-non-null-assertion */
import * as k8s from "@kubernetes/client-node";
import * as fs from "fs";

// Configure the operator to deploy your custom resources
// and the destination namespace for your pods
const MYCUSTOMRESOURCE_GROUP = "custom.example.com";
const MYCUSTOMRESOURCE_VERSION = "v1";
const MYCUSTOMRESOURCE_PLURAL = "mycustomresources";
const NAMESPACE = "workers";

// This value specifies the amount of pods that your deployment will have
interface MyCustomResourceSpec {
  size: number;
}

interface MyCustomResourceStatus {
  pods: string[];
}

interface MyCustomResource {
  apiVersion: string;
  kind: string;
  metadata: k8s.V1ObjectMeta;
  spec?: MyCustomResourceSpec;
  status?: MyCustomResourceStatus;
}

// Generates a client from an existing kubeconfig whether in memory
// or from a file.
const kc = new k8s.KubeConfig();
kc.loadFromDefault();

// Creates the different clients for the different parts of the API.
const k8sApi = kc.makeApiClient(k8s.AppsV1Api);
const k8sApiMC = kc.makeApiClient(k8s.CustomObjectsApi);
const k8sApiPods = kc.makeApiClient(k8s.CoreV1Api);

// This is to listen for events or notifications and act accordingly
// after all it is the core part of a controller or operator to
// watch or observe, compare and reconcile
const watch = new k8s.Watch(kc);

// Then this function determines what flow needs to happen
// Create, Update or Destroy?
async function onEvent(phase: string, apiObj: any) {
  log(`Received event in phase ${phase}.`);
  if (phase == "ADDED") {
    scheduleReconcile(apiObj);
  } else if (phase == "MODIFIED") {
    try {
      scheduleReconcile(apiObj);
    } catch (err) {
      log(err);
    }
  } else if (phase == "DELETED") {
    await deleteResource(apiObj);
  } else {
    log(`Unknown event type: ${phase}`);
  }
}

// Call the API to destroy the resource, happens when the CRD instance is deleted.
async function deleteResource(obj: MyCustomResource) {
  log(`Deleted ${obj.metadata.name}`);
  return k8sApi.deleteNamespacedDeployment(obj.metadata.name!, NAMESPACE);
}

// Helpers to continue watching after an event
function onDone(err: any) {
  log(`Connection closed. ${err}`);
  watchResource();
}

async function watchResource(): Promise<any> {
  log("Watching API");
  return watch.watch(
    `/apis/${MYCUSTOMRESOURCE_GROUP}/${MYCUSTOMRESOURCE_VERSION}/namespaces/${NAMESPACE}/${MYCUSTOMRESOURCE_PLURAL}`,
    {},
    onEvent,
    onDone,
  );
}

let reconcileScheduled = false;

// Keep the controller checking every 1000 ms
// If after any condition the controller needs to be stopped
// it can be done by setting reconcileScheduled to true
function scheduleReconcile(obj: MyCustomResource) {
  if (!reconcileScheduled) {
    setTimeout(reconcileNow, 1000, obj);
    reconcileScheduled = true;
  }
}

// This is probably the most complex function since it first checks if the
// deployment already exists and if it doesn't it creates the resource.
// If it does exists updates the resources and leaves early.
async function reconcileNow(obj: MyCustomResource) {
  reconcileScheduled = false;
  const deploymentName: string = obj.metadata.name!;
  // Check if the deployment exists and patch it.
  try {
    const response = await k8sApi.readNamespacedDeployment(deploymentName, NAMESPACE);
    const deployment: k8s.V1Deployment = response.body;
    deployment.spec!.replicas = obj.spec!.size;
    k8sApi.replaceNamespacedDeployment(deploymentName, NAMESPACE, deployment);
    return;
  } catch (err) {
    log("An unexpected error occurred...");
    log(err);
  }

  // Create the deployment if it doesn't exists
  try {
    const deploymentTemplate = fs.readFileSync("deployment.json", "utf-8");
    const newDeployment: k8s.V1Deployment = JSON.parse(deploymentTemplate);

    newDeployment.metadata!.name = deploymentName;
    newDeployment.spec!.replicas = obj.spec!.size;
    newDeployment.spec!.selector!.matchLabels!["deployment"] = deploymentName;
    newDeployment.spec!.template!.metadata!.labels!["deployment"] = deploymentName;
    k8sApi.createNamespacedDeployment(NAMESPACE, newDeployment);
  } catch (err) {
    log("Failed to parse template: deployment.json");
    log(err);
  }

  //set the status of our resource to the list of pod names.
  const status: MyCustomResource = {
    apiVersion: obj.apiVersion,
    kind: obj.kind,
    metadata: {
      name: obj.metadata.name!,
      resourceVersion: obj.metadata.resourceVersion,
    },
    status: {
      pods: await getPodList(`deployment=${obj.metadata.name}`),
    },
  };

  try {
    k8sApiMC.replaceNamespacedCustomObjectStatus(
      MYCUSTOMRESOURCE_GROUP,
      MYCUSTOMRESOURCE_VERSION,
      NAMESPACE,
      MYCUSTOMRESOURCE_PLURAL,
      obj.metadata.name!,
      status,
    );
  } catch (err) {
    log(err);
  }
}

// Helper to get the pod list for the given deployment.
async function getPodList(podSelector: string): Promise<string[]> {
  try {
    const podList = await k8sApiPods.listNamespacedPod(
      NAMESPACE,
      undefined,
      undefined,
      undefined,
      undefined,
      podSelector,
    );
    return podList.body.items.map((pod) => pod.metadata!.name!);
  } catch (err) {
    log(err);
  }
  return [];
}

// The watch has begun
async function main() {
  await watchResource();
}

// Helper to pretty print logs
function log(message: string) {
  console.log(`${new Date().toLocaleString()}: ${message}`);
}

// Helper to get better errors if we miss any promise rejection.
process.on("unhandledRejection", (reason, p) => {
  console.log("Unhandled Rejection at: Promise", p, "reason:", reason);
});

// Run
main();        

Introducing the deployment.json File

Let's clarify the crucial role of the deployment.json file in your TypeScript operator setup. Here's its purpose:

• Resource Specification: This file describes the exact Kubernetes resources (e.g., Pods, Deployments, Services) that your operator should create or manage in response to changes in your Custom Resource.
• Blueprint: Consider it the blueprint that the operator follows when your Custom Resource is created, updated, or deleted.

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "mycustomresource"
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "mycustomresource"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "app": "mycustomresource"
        }
      },
      "spec": {
        "containers": [
          {
            "command": ["sleep", "3600"],
            "image": "busybox:latest",
            "name": "busybox"
          }
        ]
      }
    }
  }
}        

Defining Our Custom Resource

The Custom Resource Definition (CRD) is fundamental to how we interact with our TypeScript operator. Let's break down its importance:

• Language of Communication: Through the CRD, we provide instructions to the operator. We specify the type of task we need it to manage and the relevant input it requires.
• Structure: The CRD has a predefined structure. It includes fields to capture data essential to the task (think of these like parameters you'd pass to a function).
• Trigger: When we create an instance of this Custom Resource in the Kubernetes cluster, it signals our operator to spring into action, carrying out the instructions based on the data provided.

CRD usage example:

apiVersion: custom.example.com/v1
kind: MyCustomResource
metadata:
  name: mycustomresource-sample
  namespace: workers
spec:
  size: 2        

Let's Connect the Dots

Remember the deployment.json? Imagine your operator can extract that image information from the custom resource and dynamically populate the image field in the deployment!

To grasp the full picture, I highly recommend cloning the repository and exploring how the code works. Feel free to experiment and modify the operator to deepen your understanding.

Cleaning Up Your Operator

When you're finished experimenting with the operator, follow these steps to remove it from your cluster:

? kubectl delete -f resources/mycustomresource-sample.yaml
? kustomize build resources/ | kubectl delete -f -
namespace "ts-operator" deleted
customresourcedefinition.apiextensions.k8s.io "mycustomresources.custom.example.com" deleted
serviceaccount "ts-operator" deleted
clusterrole.rbac.authorization.k8s.io "mycustomresource-editor-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "manager-rolebinding" deleted
deployment.apps "ts-operator" deleted

? kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-558bd4d5db-284q5                     1/1     Running   0          10h
kube-system          coredns-558bd4d5db-5qs64                     1/1     Running   0          10h
kube-system          etcd-kind-control-plane                      1/1     Running   0          10h
kube-system          kindnet-njtns                                1/1     Running   0          10h
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          10h
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          10h
kube-system          kube-proxy-d2gkx                             1/1     Running   0          10h
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          10h
local-path-storage   local-path-provisioner-547f784dff-tp6cq      1/1     Running   0          10h        

Wrapping Up

I hope this exploration of building Kubernetes operators with TypeScript has been insightful. If the examples from Nodeshift sparked your interest, be sure to check out their resources for further learning. Feel free to connect with me on Twitter or GitHub or Follow me here to continue the conversation!

You can find the complete source code for this tutorial here.

Disclaimer: While I didn't demonstrate specifically on OpenShift, the core concepts and techniques easily translate to standard Kubernetes clusters.