Extending K8S: Mutating Web?hooks

In this tutorial, I will be discussing as how we can extend K8S using Dynamic Admission Controller.

Before we start writing admission controller, let us revise what are admission webhooks. As per k8s documentation,

Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked.

For more details, please refer k8s official documentation.

Goal

We will be writing a Mutating Webhook to implement a pattern where we need to inject another container as side car. Say, we are running a rest-api container and we want to front that with nginx Container as sidecar in a Pod. This nginx container will be used to terminate SSL and act as reverse proxy for rest-api container. As a user, we will only be writing rest-api container manifest within Pod and let mutating webhook inject nginx to it.

Environment

1. Minikube for Kubernetes Cluster
2. Cert-Manager for managing Certificates
3. Language: Go

The code for this blog is checked in to GitHub.

Note: Most of the images in this tutorial are hyperlinked to source code in GitHub.

Directory Structure

1. src: Source code directory
2. configs: Contains k8s manifests required to create necessary resources.

Installation

Installing Kubernetes Cluster

1. Please refer here for installing minikube.
2. Start minikube server

$ minikube start        

Installing Cert-Manager

For Webhook, we need to have a CA which can sign certificates for TLS. We are using cert-manager for this. We will use cert-manager provided certificates for TLS termination at Nginx as well. To get latest version of cert-manager, please refer here.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml          

Verify that all pods of cert-manager are up

$ kubectl get pods -n cert-manager         

As mentioned in official documentation of cert-manager, it adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.

Webhook Development

A mutating webhook is HTTP callback that receive admission requests. Lets start by writing a HTTP server.

Build and Run it.

$ export CGO_ENABLED=0 go build -o webhook
$ ./webhook        

Verify, if you can access https://localhost/mutate from browser.

1. As we will receive webhook requests from Kubernetes, we need to translate those requests to an understandable format such as Objects or Struct. To do this, we need to deserialize them using K8S Universal Deserializer.

2. In our local environment, we usually access k8s using Kube Config file. However, this file will not be available within the cluster and we need to use in-cluster config to achieve same. This can be done using GetConfigOrDie function from client package. As per package definition, the config precedence is as follows:

• --kubeconfig flag pointing at a file
• KUBECONFIG environment variable pointing at a file
• In-cluster config if running in cluster
• $HOME/.kube/config if exists

Now, If we need to invoke any K8S API to get cluster information, we need a ClientSet.

3. To test, if ClientSet is working, we have written a simple API, that count total running pods in a cluster.

# ./webhook  
Total pod running in cluster: 12          

4. K8S enforce security at all steps and one such requirement is to run webhook on SSL. We now need to change our HTTP Server from Non TLS to TLS Server. We will also provide options to override TLS certificate and key location and listening port.

Later in this blog, using cert-manager, we will create a secret i.e. sidecar-injector-certs which will provide tls.crt and tls.key when mounted as volume to Webhook Pod.

5. Kubernetes sends AdmissionReview as HTTP body and expects an AdmissionResponse after evaluating AdmissionRequest. AdmissionReview is a JSON object that contains the Pod resource itself with extra metadata. In general, it needs to be passed to a Admitter and get it to generate an AdmissionReview.

This is where we can have webhook make decision whether it need to allow request, deny or make some changes to it. Request allow/deny is controlled by setting flag to Allowed in AdmissionResponse.

Lets us write function to get AdmissionReview Request, pass it to universal decoder for verification, and then extract Pod object from it to mutate.

6. Once verified, we will then extract Pod object from Admission Request

7. Now we have pod object with us. We need to start patching it. Our webhook will mutate Pod object by injecting a Label and Sidecar container. The sidecar container need ssl certificate and nginx.conf for nginx to act as reverse proxy.

Considering this, we would need

1. A Function to provide volume config
2. Another function to provide sidecar config
3. Labels are map and we can add a new key to it directly.

• To start, we will first create a struct that can hold the require patch config. It contains an array of Container and Volume.

• A Container and Volume object has various field and we created a struct to abstract the unnecessary information. This struct is called NginxSideCarConfig which take Side Car Container Name, ImageName, ImagePullPolicy, Port and VolumeMounts.

• To provide Volume definition, we created a function getPodVolumes(). It returns an array of Volumes that needs to be present in Pod for mounting in Container.

• Now, to generate sidecar config, we wrote a function generateNginxSideCarConfig(), that takes NginxSideCarConfig and array of Volumes as argument and return Config object.

• All of above is orchestrated using another function getNginxSideCarConfig() which return Config Object with Container Information having VolumeMounts and Volumes attached to Pod.

8. The function getNginxSideCarConfig() (as above) return a Config object, which now needs to be patched with Pod object that we extracted from AdmissionRequest.

patches, _ := createPatch(pod, sideCarConfig)        

PatchOperation

Before we dive what createPatch() function do, let us understand what is PatchOperation.

K8S follow RFC6902 which defines JavaScript Object Notation (JSON) Patch for expressing a sequence of operations to apply to a JSON document. To summarize, a Patch Object needs to have three items in it. These are

1. Operation (Op)
2. Path
3. Value

While patching a AdmissionRequest (JSON), K8S expect AdmissionReviewResponse (or AdmissionResponse) to contain these three items. Therefore, we created a struct which can define patch operations

In Function createPatch(), we are simply iterating over Config object and adding to to PatchOperation array.

9. Once all patching is completed and added to patch operation array, we then need to convert patches to byte slice and add it to AdmissionResponse

10. At the end, we need to send this back as HTTP response.

This conclude development of webhook source code, but hold on, it is not yet ready to be used.

Publishing Webhook to Docker

We need to create a docker image containing Webhook source code so that it can be deployed to Cluster.

While building the docker image for webhook, we use multi-stage docker build where one stage act as builder and other as image containing executable code.

For build stage, I am using golang:1.17-alpine image, and for deployable webhook container image, I prefer to use alpine:3.10

Build and Push

I have used Docker Hub as container repository

docker build . -t yks0000/sample-mutating-webhook:v2
docker push yks0000/sample-mutating-webhook:v2        

Writing Kubernetes Resources Manifests for Webhook

1. We first start with creating certificates that are required by Webhook and Nginx. As you can see in manifest below, we are first creating a cert-manager self-signed Issuer and then using it for generating Certificate.

Name: certs.yaml

2. Create a MutatingWebhookConfiguration.

Name: webhook.yaml

• here, cert-manager.io/inject-ca-from, will make cainjector populate the caBundle field using CA data from a cert-manager Certificate.
• We support v1beta1 admissionReviewVersions
• objectSelector: It help control mutation. If applied, only Pod in cluster having label defined in it will be valid for mutation. Others will be ignored.
• clientConfig: It defines, the k8s service that forward request to webhook Pod. We also need to define context path which will handle Mutating webhook request.
• rules: It define, Operations, Permission and resource on which a mutating webhook should run.

3. RBAC for Authentication and Authorization

Name: rbac.yaml

We create a Service account, a Cluster Role and Cluster Role binding to provide required API authz and authn permission to webhook. This service account will be used in deployment manifest later.

4. Webhook Deployment Manifest

Name: deployment.yaml

If you can notice, we are creating Volume Mounts from a secret named as sidecar-injector-certs. This certs are used within webhook code to start HTTP server on TLS.

5. Webhook Service Manifest

Name: service.yaml

6. Config Map

If you remember in one of the function above getPodVolumes(), we created a k8s volumes array that was attached to Pod so that it can be mounted inside Container as volume. One of the volume in it is generated using ConfigMap (hard-coded as nginx-conf)

For this, we created a nginx.conf and then used it to create CM.

kubectl create cm nginx-conf --from-file=nginx.conf=./configs/nginx.conf        

Creating K8S Resources

• Create Certificate for Webhook?

$ kubectl apply -f configs/certs.yaml

issuer.cert-manager.io/selfsigned-issuer unchanged
certificate.cert-manager.io/sidecar-injector-certs unchanged        

• Deploy RBAC

$ kubectl apply -f configs/rbac.yaml

serviceaccount/sample-mutating-webhook created
clusterrole.rbac.authorization.k8s.io/sample-mutating-webhook created
clusterrolebinding.rbac.authorization.k8s.io/sample-mutating-webhook created?        

• Create Deployment

$ kubectl apply -f configs/deployment.yaml
deployment.apps/sample-mutating-webhook created        

• Deploy Service

$ kubectl apply -f configs/service.yaml
service/sample-mutating-webhook created?        

• Deploy Webhook

$ kubectl apply -f configs/webhook.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/sample-mutating-webhook created        

• Verify Pods

$ kubectl -n default get pods | grep sample-mutating-webhook
sample-mutating-webhook-5d8666ffc7-4ljdh 1/1???? Running?? 0????????? 39s        

Check Logs of pod, should emit log showing total number of pods

$ kubectl logs sample-mutating-webhook-5d8666ffc7-4ljdh
Total pod running in cluster: 13        

Test

In our example, we are adding a label nginx-sidecar to pod definition and injecting nginx container as sidecar before API server sent it to controller to schedule it.?

As we also added objectSelector to webhook.yaml, we need to make sure inject-nginx-sidecar: "true" label is added to pod definition, otherwise our mutating webhook will ignore the request.?

Lets create a example Pod, and before that, let us review its manifest

Name: example-pod.yaml

We can see that we have only one Container which runs echoserver by exposing port 8080. As we want to inject nginx side car, we added a label inject-nginx-sidecar: "true" to it.

Lets us now create it

$ kubectl apply -f configs/example-pod.yaml        

In screenshots below we can see that 2 container are getting created though in our manifest we had only one. The other one is nginx.

It has also applied the extra label that we injected from webhook.

HTTP Test

Let us test by hitting the pod via Nginx. The request should first go to Nginx and then to rest-api container. As we are running nginx with TLS, we will be sending request on HTTPS.

We need to expose Pod and create tunnel (require for minikube k8s cluster to access LoadBalancer service on Host machine)

We will use stern to tail multi-container log for verification

In a terminal, run command

?stern example-pod        

In another terminal, run below command to expose port 443 (Nginx Port) of Pod.

kubectl expose pod example-pod --type=LoadBalancer --port=443        

then, open minikube tunnel. For 443 port, you need to enter password as on macOS, this is privileged port.

Send a request now:

$ curl https://127.0.0.1 -k

GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.79.1
Accept: */*
Connection: close
Hostname: example-pod
Time: 2022-06-01 12:24:00.6134619 +0000 UTC m=+871.490196101
X-Forwarded-For: 172.17.0.1        

Verify the logs:

We can see that the request has reached both container. rest-api being backend has logged it before nginx.

This conclude the blog (tutorial). The code and config file are present in GitHub.