Creating Custom Kubernetes Event?Watcher

Kubernetes triggers events whenever there is a state change within your cluster. These are stored somewhere in the cluster and have a limited retention time, by default 1 hour. These events provide information that can supplement debugging in the case of deployed resource failing.

You can extend the TTL from 1 hour using the tag referenced here --event-ttl but some cloud providers do not provide a way to natively do this. Hence my desire to create a custom event watcher that is customizable and can be deployed to any cluster.

I created my own app, kubernetes-watcher, that I can deploy to my cluster and it will inform me of any events that I configure it to watch. The app is available on GitHub if you would like to check it out https://github.com/mamin11/kubernetes-watcher.

Currently, this allows me to configure what events to watch through kubernetes configmaps. These are then injected into the app at start time and will trigger an alert if a matching event if found within the cluster. This config looks something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filter-config
data:
  config.json: |
    [
      {
        "kind": "Pod",
        "reasons": ["Killing", "Failed", "FailedScheduling", "BackOff", "Unhealthy"]
      }
    ]        

The above will add a filter to watch events that happen on Pods and the reason for the event is in the list provided. We can add as many filters as we like just by adding a new filter object to the array.

The app is able to do this by making use of the kubernetes API and we combine this with a service account with the following role

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-event-watcher-cluster-role
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]        

In addition, we send triggered events through configurable message channel. Currently, there is support for sending events via email and/or slack. These channels can also be configured through configmaps as shown below. By default all channels are disabled.

apiVersion: v1
kind: ConfigMap
metadata:
  name: k8s-watcher-config
data:
  ...
  notification.slack.config.enabled: "true"
  notification.slack.config.channel: "k8s-events"
  notification.slack.config.username: "Kubernetes-Watcher"
  notification.email.config.enabled: "true"
  notification.email.config.host: "host.docker.internal"
  notification.email.config.port: "1025"
  notification.email.config.protocol: "smtp"
  notification.email.config.to: "[email protected]"        

These is test pods available for you to deploy in?.test-pods directory of the the GitHub repo.

Happy coding.