Kubernetes | Extending the functionality of your cluster with CRDs and Java Controllers

TL;DR

Kubernetes is like any traditional application we develop every day, some resources related together in a way or another with a database which is "etcd" for Kubernetes to store these information, however in Kubernetes "as a state-full application" there is one extra component known as controllers.

Each controller is responsible for watching a particular resource's "crud" operations and updating the current cluster state according to the desired state declared in this resource or according to the operation being made to it, for example, the pod resource has its own controller which responsible for pulling the container/s image/s and run them, the same concept with services, deployments or any other Kubernetes resource.

In this article i'll try to illustrate how can we define a custom resource and write a controller for it to introduce a new functionality in the cluster.

The official CRDs docs

Why do we need to write a custom resource & a controller ?

1. To extend the functionality of an existing resource
2. Or to replacing existing components
3. Or even to add new concepts / functionalities to the cluster

"without modifying the Kubernetes codebase".

The problem/s

Assuming that there is a team that specialized in provisioning managed services for its clients on kubernetes, Mysql clusters or Redis clusters. This team writes the same config files (kubernetes yaml) everyday with some minor differences between them.

Or another team that works on a microservices application "which are mostly written in the same technology and have the same configuration interface". They also write the same config files to get these apps running on Kubernetes with every new service.

The CRD solution

For both of them wouldn't it be much better if they could abstract the common desired operations from theses config files they write every now and then in a different application to do the job on their behalf. Operations like running a pod, maintaining X number of replicas with some specific configuration etc.

The first team need to define a new Kubernetes resource/kind let's call it "MySQL" with some specs that describe the number of the desired nodes, maybe the storage size or any other configuration.

The second team who is working on a product called "LinkedIn" for example, can also create a new resource/kind called "LinkedIn" with some specs like the image that represents the service "users-service" or "posts-service" and the desired replicas along with some environment variables to configure the application service.

Coding time

I'll try to help the second team in this demo, the resource i'll define is for a product called "Fidelyo" which is a microservices application all written in Java. The service is very simple, it basically starts a vertx server on port 9090 that reads "MESSAGE" environment variable and returns it on the "/" endpoint.

Fidelyo service

The code in the service is just like this

public class Main {

    public static void main(String[] args) {

        Vertx vertx = Vertx.vertx();

        HttpServer server = vertx.createHttpServer();

        Router router = Router.router(vertx);

        router.route().handler(context -> {

            HttpServerResponse response = context.response();
            response.putHeader("content-type", "text/plain");

            String message = System.getenv("MESSAGE");
            if (message == null || "".equals(message))
                message = "Hello World from Vert.x-Web!";

            response.end(message);

        });

        server.requestHandler(router).listen(9090);

    }

}

add vertx maven dependency

<dependency>
    <groupId>io.vertx</groupId>
    <artifactId>vertx-web</artifactId>
    <version>${vertx-version}</version>
</dependency>

and finally add jib to build and push the container for you

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>${jib-maven.version}</version>
    <configuration>
        <to>
            <image>${jib-to-image}</image>
            <auth>
                <username>${jib-to-auth-username}</username>
                <password>${jib-to-auth-password}</password>
            </auth>
        </to>
        <from>
            <image>${jib-from-image}</image>
        </from>
    </configuration>
    <executions>
        <execution>
            <id>build</id>
            <goals>
                <goal>build</goal>
            </goals>
            <phase>package</phase>
        </execution>
    </executions>
</plugin>

Fidelyo resource & controller

The devops engineer at fidelyo needs to write a deployment file and expose it as a "maybe" LoadBalancer service then add the common environment variables with every new service.

My goal is to allow him to apply a new yaml file to the cluster that looks like this, to get all the job done for him.

apiVersion: bishoybasily.gmail.com/v1
kind: Fidelyo
metadata:name: <<Name>>
  namespace: production
spec:
  image: <<DockerImageReference>>:<<ImageTAG>>
  port: 8080
  replicas: <<NumberOfReplicas>>
  serviceType: LoadBalancer
  envVars:
    - name: <<KEY>>
      value: <<VALUE>>

Behind the scenes the controller will notice that a fidelyo resource got created and accordingly, it will create a deployment with the provided image, specify it's replicas, apply the probes, define it's update strategy and finally expose it as the specified service type that targets the specified port on the pod.

So, how to tell Kubernetes that we want to introduce this new kind!

The API-Resources

kubectl api-resources

Will list the available list of resources, pipe the output to grep and make sure that there is nothing called "fidelyo" in the list

kubectl api-resources | grep -i "fidelyo"

Now let's define this resource

Custom resource definitions

There are two ways for adding a new resource definition to the cluster

• The declarative way - through config files (kubectl is responsible for doing the imperative part in this case)
• The imperative way - through the apiserver directly

Keep in mind that defining the resource doesn't mean that the desired functionality is added, the same idea for a cluster that doesn't have an ingress controller. Applying ingress resource to this cluster will do nothing until having an ingress controller on the cluster to understand what does ingress mean in terms of updating the cluster state "adding a routing functionality"

Fidelyo resource definition, the declarative way

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: fidelyos.bishoybasily.gmail.com
spec:
  group: bishoybasily.gmail.com
  versions:
    - name: v1
      served: true
      storage: true
  scope: Namespaced
  names:
    plural: fidelyos
    singular: fidelyo
    kind: Fidelyo
    shortNames:
      - fd
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            image:
              type: string
            replicas:
              type: integer
            port:
              type: integer
            serviceType:
              type: string
            envVars:
              type: object

Please note those parts, the name "fidelyos.bishoybasily.gmail.com", the group "bishoybasily.gmail.com", the names section including the kind "Fidelyo", and the most important part the schema properties, as you can see the properties starts with spec "to stay aligned with the naming convention", the spec property is an object and has sub-properties, image -> string, replicas -> integer, port -> integer -> serviceType -> string, envVars -> object, those spec sub-properties will inform the controller how to create the deployment and the service for any "Fidelyo" kind.

Fidelyo resource definition, the imperative way

As I mentioned Kubernetess like any other application has an API interface to interact with its components and luckily it is a Restful HTTP interface called kube-apiserver. To simplify the interactions with it i'll be using the superior fabric8 client to interact with it.

Before getting started, let's make sure that the .kube/config file is populated with the right information. If you are using microk8s for development, export its config to the .kube/config file "

This will overwrite the config file if any

microk8s.kubectl config view --raw > $HOME/.kube/config

Let's start with a maven Java app, add the fabric8 client dependency

<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>kubernetes-client</artifactId>
    <version>${fabric8-client.version}</version>
</dependency>

To validate the setup, write a basic client to list the pods names in current namespace.

try (KubernetesClient client = new DefaultKubernetesClient()) {
            client
                    .pods()
                    .list()
                    .getItems()
                    .stream()
                    .map(Pod::getMetadata)
                    .map(ObjectMeta::getName)
                    .forEach(System.out::println);
}

If this runs successfully, then let's define our resource

private CustomResourceDefinition getOrCreateTheDefinition(KubernetesClient client) {

   
    CustomResourceDefinitionList crds = client.customResourceDefinitions().list();

    // list all the crds in the cluster

    List<CustomResourceDefinition> crdsItems = crds.getItems();


    CustomResourceDefinition fidelyoCRD = null;

    // try to find the targetted crd

    for (CustomResourceDefinition crd : crdsItems) {
        ObjectMeta metadata = crd.getMetadata();
        if (metadata != null) {
            String name = metadata.getName();
            if (CRD_NAME.equals(name)) {
                fidelyoCRD = crd;
                // return the crd if found, then no need for creating it
            }
        }
    }

    // if the crd wasn't found "which means that this is the first time we run this"

    if (fidelyoCRD == null) {

        // create it, this is the equivalent Java code for the declaritive way mentioned earlier

        fidelyoCRD = new CustomResourceDefinitionBuilder()
                // @formatter:off
                .withApiVersion(CRD_API_VERSION)
                .withNewMetadata()
                    .withName(CRD_NAME)
                .endMetadata()
                .withNewSpec()
                    .withGroup(CRD_GROUP)
                    .withVersion(CRD_VERSION)
                    .withScope(CRD_SCOPE)
                    .withNewNames()
                        .withKind(CRD_KIND)
                        .withShortNames(CRD_SHORT_NAME)
                        .withPlural(CRD_PLURAL)
                    .endNames()
                .endSpec()
                // @formatter:on
                .build();

        client.customResourceDefinitions().create(fidelyoCRD);

    }

    return fidelyoCRD;

}

and here are the constants

private static String
        CRD_API_VERSION = "apiextensions.k8s.io/v1beta1",
        CRD_GROUP = "bishoybasily.gmail.com",
        CRD_NAME = "fidelyos." + CRD_GROUP,
        CRD_SCOPE = "Namespaced",
        CRD_VERSION = "v1",
        CRD_KIND = "Fidelyo",
        CRD_SHORT_NAME = "fd",
        CRD_PLURAL = "fidelyos";

Back to the client part

try (KubernetesClient client = new DefaultKubernetesClient()) {

    CustomResourceDefinition fidelyoCRD = getOrCreateTheDefinition(client);
}

Now let's validate that the resources have the new definition, again run

kubectl api-resources | grep -i "fidelyo"

The output now should be different

Now the resource itself and the spec part.

Fabric8 client requires you to extend the following classes in order to build a dedicated client for interacting with this particular resource.

// the custom kind/Resource
@Data
@EqualsAndHashCode(callSuper = true)
@Accessors(chain = true)
public class Fidelyo extends CustomResource {

    private FidelyoSpec spec;

}


// the kind specs we mentioned earlier,
// please ignore the deployment and the service for now, i'll come to this later
@Data
@Accessors(chain = true)
@JsonDeserialize
public class FidelyoSpec implements KubernetesResource {

    private String image;
    private Integer replicas;
    private Integer port;
    private String serviceType;
    private Set<EnvVar> envVars = new HashSet<>();

    private Service service;
    private Deployment deployment;

}


// the doneable, which will be returned as a result for the create operation for example, 
// doneable contains a done method that will return the Fidelyo item after applying it to the cluster
public class FidelyoDoneable extends CustomResourceDoneable<Fidelyo> {

    public FidelyoDoneable(Fidelyo resource, Function function) {
        super(resource, function);
    }

}

// the list, which will be returned as a result for operations like get fidelyos,
// it contains a method called getList which should return an iterable of the Fidelyos available in the cluster,
// it also holds some extra information about the list itself
public class FidelyoList extends CustomResourceList<Fidelyo> {
    
}

after declaring those POJOs, we should be able to build a client for this custom resource

private NonNamespaceOperation<Fidelyo, FidelyoList, FidelyoDoneable, Resource<Fidelyo, FidelyoDoneable>> buildTheCrdClient(KubernetesClient client, CustomResourceDefinition fidelyoCRD) {
    return client.customResources(fidelyoCRD, Fidelyo.class, FidelyoList.class, FidelyoDoneable.class).inNamespace(client.getNamespace());
}

Back to the client part

try (KubernetesClient client = new DefaultKubernetesClient()) {

    CustomResourceDefinition fidelyoCRD = getOrCreateTheDefinition(client);

    NonNamespaceOperation<Fidelyo, FidelyoList, FidelyoDoneable, Resource<Fidelyo, FidelyoDoneable>> fidelyoClient = buildTheCrdClient(client, fidelyoCRD);
}

Awesome, we have created the fidelyoCRD and its client, now let's start the real controller job which is to keep watching the changes being made to this resource and behave accordingly.

Our controller is interested in three main events (ADD, MODIFY and DELETE), as i mentioned when a new fidelyo applied to the cluster the controller should grab its specs and create a new deployment and a service for this application that matches the desired specs, also if the watcher noticed that some fidelyo was removed, it should also remove its associated deployment and service as well.

  private void watchTheResourceChanges(KubernetesClient client, NonNamespaceOperation<Fidelyo, FidelyoList, FidelyoDoneable, Resource<Fidelyo, FidelyoDoneable>> fidelyoClient) {

    fidelyoClient.watch(new Watcher<Fidelyo>() {

        @Overridepublic void eventReceived(Action action, Fidelyo resource) {

            FidelyoSpec spec = resource.getSpec();
            ObjectMeta metadata = resource.getMetadata();
            String ns = resource.getMetadata().getNamespace();

            if (action.equals(Action.ADDED) || action.equals(Action.MODIFIED)) {

                // @formatter:off
                Deployment deployment = new DeploymentBuilder(true)
                        .withNewMetadata()
                            .withNamespace(metadata.getNamespace())
                            .withName(metadata.getName()).withNamespace(ns)
                        .endMetadata()
                        .withNewSpec()
                            .withReplicas(spec.getReplicas())
                            .withNewTemplate()
                                .withNewMetadata()
                                    .withNamespace(metadata.getNamespace())
                                    .addToLabels("app", metadata.getName())
                                .endMetadata()
                                .withNewSpec()
                                    .addNewContainer()
                                        .withName(metadata.getName())
                                        .withImage(spec.getImage())
                                        .addNewPort()
                                            .withContainerPort(spec.getPort())
                                        .endPort()
                                        .addAllToEnv(spec.getEnvVars())
                                    .endContainer()
                                .endSpec()
                            .endTemplate()
                            .withNewSelector()
                                .addToMatchLabels("app", metadata.getName())
                            .endSelector()
                        .endSpec()
                        .build();
                        // @formatter:on

                deployment = client.extensions().deployments().inNamespace(ns).createOrReplace(deployment);

                // @formatter:off
                Service service = new ServiceBuilder(true)
                        .withNewMetadata()
                            .withNamespace(metadata.getNamespace())
                            .withName(metadata.getName() ).withNamespace(ns)
                        .endMetadata()
                        .withNewSpec()
                            .addToSelector("app", metadata.getName())
                            .withType(spec.getServiceType())
                            .addNewPort()
                                .withNewTargetPort(spec.getPort())
                                .withPort(80)
                                .withName("default")
                                .withProtocol("TCP")
                            .endPort()
                        .endSpec()
                        .build();
                        // @formatter:on

                service = client.services().inNamespace(ns).createOrReplace(service);

                if (action.equals(Action.ADDED))
                    fidelyoClient.createOrReplace(resource.setSpec(spec.setDeployment(deployment).setService(service)));

            }

            if (action.equals(Action.DELETED)) {
                client.extensions().deployments().inNamespace(ns).delete(resource.getSpec().getDeployment());
                client.services().inNamespace(ns).delete(resource.getSpec().getService());
            }


        }

        @Overridepublic void onClose(KubernetesClientException cause) {

        }

    });
}

As you can see, fabric8 client has a very simple interface to register a watcher for a specific resource, on the eventReceived method, we should check if the event of type ADDED or MODIFIED to update the deployment and the service with new desired specs and only if the event is ADDED update the resource itself with the deployment and the service.

For the DELETED event delete the deployment and the service we assigned earlier to this resource.

Back to the client part

try (KubernetesClient client = new DefaultKubernetesClient()) {

    CustomResourceDefinition fidelyoCRD = getOrCreateTheDefinition(client);

    NonNamespaceOperation<Fidelyo, FidelyoList, FidelyoDoneable, Resource<Fidelyo, FidelyoDoneable>> fidelyoClient = buildTheCrdClient(client, fidelyoCRD);

    watchTheResourceChanges(client, fidelyoClient);

    
    Thread.currentThread().join(); // keep the app runnig


}

Now your controller is ready to understand a resource like this

apiVersion: bishoybasily.gmail.com/v1
kind: Fidelyo
metadata:
  name: fidelyo-service-x
  namespace: fcrd
spec:
  image: bishoybasily/service:fcrd-1.2
  port: 9090
  replicas: 2
  serviceType: LoadBalancer
  envVars:
    - name: MESSAGE
      value: Java loves kubernetes - service X

By applying this resource to the cluster, a new deployment named "fidelyo-service-x" with 2 replicas will be added along with a LoadBalancer service targets 9090 on the deployment pod.

Finally let's add jib's maven plugin to the controller application in order to be able to deploy it to the cluster

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>${jib-maven.version}</version>
    <configuration>
        <to>
            <image>${jib-to-image}</image>
            <auth>
                <username>${jib-to-auth-username}</username>
                <password>${jib-to-auth-password}</password>
            </auth>
        </to>
        <from>
            <image>${jib-from-image}</image>
        </from>
    </configuration>
    <executions>
        <execution>
            <id>build</id>
            <goals>
                <goal>build</goal>
            </goals>
            <phase>package</phase>
        </execution>
    </executions>
</plugin>

If you tried to deploy the controller as a deployment or a pod it should fail because of the missing permissions, the default service account is not authorized to talk to the kube-apiserver the way we want, that is why we need to create a custom ClusterRole and ClusterRoleBinding for a new ServiceAccount and assign it to the controller pod for this to work.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: fcrd
  name: fidelyo-crd-serviceaccount
  labels:
    type: fcrd


---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    type: fcrd
  name: fidelyo-crd-clusterrole
rules:
  - apiGroups: ["*"]
    resources: ["pods", "replicasets", "deployments", "services", "fidelyos", "customresourcedefinitions"]
    verbs: ["get", "watch", "list", "delete", "create", "update", "patch"]


---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  namespace: fcrd
  name: fidelyo-crd-clusterrolebinding
  labels:
    type: fcrd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fidelyo-crd-clusterrole
subjects:
  - kind: ServiceAccount
    name: fidelyo-crd-serviceaccount
    namespace: fcrd
    apiGroup: ""


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fidelyo-crd
  namespace: fcrd
  labels:
    type: fcrd
spec:
  replicas: 1
  selector:
    matchLabels:
      type: fcrd
  template:
    metadata:
      name: fidelyo-crd
      namespace: fcrd
      labels:
        type: fcrd
    spec:
      serviceAccount: fidelyo-crd-serviceaccount
      containers:
        - image: registry.hub.docker.com/bishoybasily/crd:fcrd-1.10
          name: fidelyo-crd

Now deploy the controller then apply the fidelyo resource file, you should get the deployment and a service created automatically and also removed automatically when you remove it.

The full source-code is available here on Github

Thanks for reading :)

Your feedback would be highly appreciated