Extending K8S: Mutating Web?hooks
In this tutorial, I will be discussing as how we can extend K8S using Dynamic Admission Controller.
Before we start writing admission controller, let us revise what are admission webhooks. As per k8s documentation,
Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked.
For more details, please refer k8s official documentation.
Goal
We will be writing a Mutating Webhook to implement a pattern where we need to inject another container as side car. Say, we are running a rest-api container and we want to front that with nginx Container as sidecar in a Pod. This nginx container will be used to terminate SSL and act as reverse proxy for rest-api container. As a user, we will only be writing rest-api container manifest within Pod and let mutating webhook inject nginx to it.
Environment
The code for this blog is checked in to GitHub.
Note: Most of the images in this tutorial are hyperlinked to source code in GitHub.
Directory Structure
Installation
Installing Kubernetes Cluster
$ minikube start
Installing Cert-Manager
For Webhook, we need to have a CA which can sign certificates for TLS. We are using cert-manager for this. We will use cert-manager provided certificates for TLS termination at Nginx as well. To get latest version of cert-manager, please refer here.
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
Verify that all pods of cert-manager are up
$ kubectl get pods -n cert-manager
As mentioned in official documentation of cert-manager, it adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.
Webhook Development
A mutating webhook is HTTP callback that receive admission requests. Lets start by writing a HTTP server.
Build and Run it.
$ export CGO_ENABLED=0 go build -o webhook
$ ./webhook
Verify, if you can access https://localhost/mutate from browser.
2. In our local environment, we usually access k8s using Kube Config file. However, this file will not be available within the cluster and we need to use in-cluster config to achieve same. This can be done using GetConfigOrDie function from client package. As per package definition, the config precedence is as follows:
Now, If we need to invoke any K8S API to get cluster information, we need a ClientSet.
3. To test, if ClientSet is working, we have written a simple API, that count total running pods in a cluster.
# ./webhook
Total pod running in cluster: 12
4. K8S enforce security at all steps and one such requirement is to run webhook on SSL. We now need to change our HTTP Server from Non TLS to TLS Server. We will also provide options to override TLS certificate and key location and listening port.
Later in this blog, using cert-manager, we will create a secret i.e. sidecar-injector-certs which will provide tls.crt and tls.key when mounted as volume to Webhook Pod.
5. Kubernetes sends AdmissionReview as HTTP body and expects an AdmissionResponse after evaluating AdmissionRequest. AdmissionReview is a JSON object that contains the Pod resource itself with extra metadata. In general, it needs to be passed to a Admitter and get it to generate an AdmissionReview.
This is where we can have webhook make decision whether it need to allow request, deny or make some changes to it. Request allow/deny is controlled by setting flag to Allowed in AdmissionResponse.
Lets us write function to get AdmissionReview Request, pass it to universal decoder for verification, and then extract Pod object from it to mutate.
6. Once verified, we will then extract Pod object from Admission Request
7. Now we have pod object with us. We need to start patching it. Our webhook will mutate Pod object by injecting a Label and Sidecar container. The sidecar container need ssl certificate and nginx.conf for nginx to act as reverse proxy.
Considering this, we would need
8. The function getNginxSideCarConfig() (as above) return a Config object, which now needs to be patched with Pod object that we extracted from AdmissionRequest.
patches, _ := createPatch(pod, sideCarConfig)
PatchOperation
Before we dive what createPatch() function do, let us understand what is PatchOperation.
K8S follow RFC6902 which defines JavaScript Object Notation (JSON) Patch for expressing a sequence of operations to apply to a JSON document. To summarize, a Patch Object needs to have three items in it. These are
While patching a AdmissionRequest (JSON), K8S expect AdmissionReviewResponse (or AdmissionResponse) to contain these three items. Therefore, we created a struct which can define patch operations
In Function createPatch(), we are simply iterating over Config object and adding to to PatchOperation array.
9. Once all patching is completed and added to patch operation array, we then need to convert patches to byte slice and add it to AdmissionResponse
10. At the end, we need to send this back as HTTP response.
This conclude development of webhook source code, but hold on, it is not yet ready to be used.
Publishing Webhook to Docker
We need to create a docker image containing Webhook source code so that it can be deployed to Cluster.
While building the docker image for webhook, we use multi-stage docker build where one stage act as builder and other as image containing executable code.
For build stage, I am using golang:1.17-alpine image, and for deployable webhook container image, I prefer to use alpine:3.10
Build and Push
I have used Docker Hub as container repository
docker build . -t yks0000/sample-mutating-webhook:v2
docker push yks0000/sample-mutating-webhook:v2
Writing Kubernetes Resources Manifests for Webhook
Name: certs.yaml
2. Create a MutatingWebhookConfiguration.
Name: webhook.yaml
3. RBAC for Authentication and Authorization
Name: rbac.yaml
We create a Service account, a Cluster Role and Cluster Role binding to provide required API authz and authn permission to webhook. This service account will be used in deployment manifest later.
4. Webhook Deployment Manifest
Name: deployment.yaml
If you can notice, we are creating Volume Mounts from a secret named as sidecar-injector-certs. This certs are used within webhook code to start HTTP server on TLS.
5. Webhook Service Manifest
Name: service.yaml
6. Config Map
If you remember in one of the function above getPodVolumes(), we created a k8s volumes array that was attached to Pod so that it can be mounted inside Container as volume. One of the volume in it is generated using ConfigMap (hard-coded as nginx-conf)
For this, we created a nginx.conf and then used it to create CM.
kubectl create cm nginx-conf --from-file=nginx.conf=./configs/nginx.conf
Creating K8S Resources
$ kubectl apply -f configs/certs.yaml
issuer.cert-manager.io/selfsigned-issuer unchanged
certificate.cert-manager.io/sidecar-injector-certs unchanged
$ kubectl apply -f configs/rbac.yaml
serviceaccount/sample-mutating-webhook created
clusterrole.rbac.authorization.k8s.io/sample-mutating-webhook created
clusterrolebinding.rbac.authorization.k8s.io/sample-mutating-webhook created?
$ kubectl apply -f configs/deployment.yaml
deployment.apps/sample-mutating-webhook created
$ kubectl apply -f configs/service.yaml
service/sample-mutating-webhook created?
$ kubectl apply -f configs/webhook.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/sample-mutating-webhook created
$ kubectl -n default get pods | grep sample-mutating-webhook
sample-mutating-webhook-5d8666ffc7-4ljdh 1/1???? Running?? 0????????? 39s
Check Logs of pod, should emit log showing total number of pods
$ kubectl logs sample-mutating-webhook-5d8666ffc7-4ljdh
Total pod running in cluster: 13
Test
In our example, we are adding a label nginx-sidecar to pod definition and injecting nginx container as sidecar before API server sent it to controller to schedule it.?
As we also added objectSelector to webhook.yaml, we need to make sure inject-nginx-sidecar: "true" label is added to pod definition, otherwise our mutating webhook will ignore the request.?
Lets create a example Pod, and before that, let us review its manifest
Name: example-pod.yaml
We can see that we have only one Container which runs echoserver by exposing port 8080. As we want to inject nginx side car, we added a label inject-nginx-sidecar: "true" to it.
Lets us now create it
$ kubectl apply -f configs/example-pod.yaml
In screenshots below we can see that 2 container are getting created though in our manifest we had only one. The other one is nginx.
It has also applied the extra label that we injected from webhook.
HTTP Test
Let us test by hitting the pod via Nginx. The request should first go to Nginx and then to rest-api container. As we are running nginx with TLS, we will be sending request on HTTPS.
We need to expose Pod and create tunnel (require for minikube k8s cluster to access LoadBalancer service on Host machine)
We will use stern to tail multi-container log for verification
In a terminal, run command
?stern example-pod
In another terminal, run below command to expose port 443 (Nginx Port) of Pod.
kubectl expose pod example-pod --type=LoadBalancer --port=443
then, open minikube tunnel. For 443 port, you need to enter password as on macOS, this is privileged port.
Send a request now:
$ curl https://127.0.0.1 -k
GET / HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.79.1
Accept: */*
Connection: close
Hostname: example-pod
Time: 2022-06-01 12:24:00.6134619 +0000 UTC m=+871.490196101
X-Forwarded-For: 172.17.0.1
Verify the logs:
We can see that the request has reached both container. rest-api being backend has logged it before nginx.
This conclude the blog (tutorial). The code and config file are present in GitHub.