Monitoring Rancher cluster health with kube-state-metrics

For several years I have been a happy user of Rancher Labs' Rancher. At several of my jobs it has been a critical component of that company's container platform(s); most frequently used for its centralized user authentication/authorization and its support for external identity providers. With Rancher, users log in once, and then, using either a web UI or CLI tools, they can manage multiple Kubernetes clusters. Rancher recommends installing Rancher in its own, dedicated, Kubernetes cluster. Other clusters that are managed by Rancher are called downstream clusters. In each of these downstream clusters, Rancher deploys a lightweight agent: the cattle-cluster-agent. When using a client such as kubectl, Kubernetes API requests are sent to the rancher API, which is then proxied to the cattle-cluster-agent. The cattle-cluster-agent then sends the request to the downstream cluster's kube-apiserver, impersonating the original user's identity.

What happens if the cattle-cluster-agent is unavailable? The downstream cluster's Kubernetes API will no longer be available to users that connect through Rancher. As an engineer, I would like to be notified whenever a downstream cluster is no longer available so that I can restore access to the Kubernetes API for the users. As far as I am aware, Rancher's out-of-the-box monitoring does not emit prometheus metrics that explain the status of downstream clusters. Regardless, Rancher's native monitoring and alerting is an opinionated solution that I might not want to lock myself into using. Preferably, I would like to be able to collect a Prometheus metric that explains the state of each downstream cluster. Based on these metrics I could then create my own Prometheus alerts.

The state of clusters can be inspected in the Rancher cluster. For each cluster, a Kubernetes resource is created of Kind cluster in Group management.cattle.io:

? kubectl get clusters.management.cattle.io


NAME           AGE
c-m-jtvzgsg2?? 24m
c-m-kc88zvgd?? 42m
local????????? 71m


? kubectl get clusters.management.cattle.io c-m-jtvzgsg2 -o yaml


apiVersion: management.cattle.io/v3
kind: Cluster
metadata:
  name: c-m-jtvzgsg2
spec:
  displayName: bazqux
  # ...
status:
  conditions:
  - lastUpdateTime: "2022-10-12T13:14:17Z"
    status: "True"
    type: Ready
  - lastUpdateTime: "2022-10-12T13:12:46Z"
    status: "True"
    type: BackingNamespaceCreated
  - lastUpdateTime: "2022-10-12T13:12:46Z"
    status: "True"
    type: DefaultProjectCreated
  # ...
        

What happens when the cattle-cluster-agent of this cluster stops running?

The cluster is unavailable! This is also reflected in the status conditions of the cluster:

? kubectl get clusters.management.cattle.io c-m-jtvzgsg2 -o json | jq '.status.conditions[] | select(.type=="Ready")'

{
? "lastUpdateTime": "2022-10-12T13:27:21Z",
? "message": "Cluster agent is not connected",
? "reason": "Disconnected",
? "status": "False",
? "type": "Ready"
}'        

To learn more about status conditions, I recommend reading this excellent article: https://maelvls.dev/kubernetes-conditions/ . For the sake of this demonstration, the TLDR is:

• The Ready type is a reliable indicator that the resource is healthy or not.
• Each condition can have one of three statuses: True, False, or Unknown.

So how do we transform status conditions into Prometheus metrics? It turns out that with the addition of a new feature called Custom Resource State Metrics recently implemented in the kube-state-metrics project, this becomes trivially easy: PR1 PR2. The official documentation can be found here: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/customresourcestate-metrics.md

I previously used the prometheus-operator to install Prometheus in my Rancher cluster. Let's install kube-state-metrics to collect custom metrics of each cluster's status conditions. kube-state-metrics is often installed as part of the excellent kube-prometheus-stack project. However, for demonstration purposes, I prefer to install kube-state-metrics separately.

? cat values.yaml

extraArgs:
  - --custom-resource-state-config
  - |
    spec:
      resources:
        - groupVersionKind:
            group: management.cattle.io
            kind: Cluster
            version: v3
          metricNamePrefix: rancher
          metrics:
            - name: cluster_status_condition
              help: Rancher cluster status condition
              labelsFromPath:
                cluster_id: [metadata, name]
                cluster_name: [spec, displayName]
              each:
                type: StateSet
                stateSet:
                  path: [status, conditions]
                  labelsFromPath:
                    type: [type]
                  labelName: status
                  valueFrom: [status]
                  list: ["True", "False", "Unknown"]


# Grant KSM permission to read the cluster resources
rbac:
  extraRules:
    - apiGroups:
        - management.cattle.io
      resources:
        - clusters
      verbs:
        - list
        - watch


# We don't let KSM emit any other metrics
collectors:
  - clusters


prometheus:
  monitor:
    enabled: true


? helm upgrade \
  --install \
  rancher-kube-state-metrics \
  prometheus-community/kube-state-metrics \
  --values values.yaml \
  --namespace kube-system


Release "rancher-kube-state-metrics" does not exist. Installing it now.
NAME: rancher-kube-state-metrics
LAST DEPLOYED: Wed Oct 12 15:46:22 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:         

After a while we can see Prometheus has started scraping our metrics:

The flag --custom-resource-state-config instructs kube-state-metrics to create a new metric called rancher_cluster_status_condition. The cluster_id and cluster_name are extracted from each cluster resource and exposed as a Prometheus label. Then we create a StateSet metric for each status condition: For each type, we create three timeseries, one for each possible value of status: True, False, or Unknown. The timeseries that represent the actual current status will have a value of 1. The others will have a value of 0.

Here you can see that type=Ready,status=True is 1 for the healthy foobar-cluster. Yet type=Ready,Status=False is 1 for the unhealthy bazqux-cluster.

We can also query all current status conditions:

Let's create a Prometheus alert that fires whenever type=Ready,status!=True == 1:

? cat prometheusrule.yaml


apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: rancher-kube-state-metrics
  namespace: kube-system
spec:
  groups:
    - name: rancher-kube-state-metrics
      rules:
        - alert: RancherClusterNotReady
          annotations:
            message: |
              Rancher cluster {{ $labels.cluster_name }} / {{ $labels.cluster_id }} has been unhealthy for at least 5 minutes.
          expr: |
            max by (cluster_name, cluster_id, type, status) (rancher_cluster_status_condition{type="Ready",status!="True"}) == 1
          for: 5m
          labels:
            severity: critical



? kubectl apply -f prometheusrule.yaml


prometheusrule.monitoring.coreos.com/rancher-kube-state-metrics created        

Et voila! We will now receive an alert whenever a cluster is not ready.

After restoring the cattle-cluster-agent the alert will resolve itself. We can see for how long the cluster had been unhealthy:

As we can see, the Custom Resource State Metrics of kube-state-metrics is a powerful feature to monitor virtually any resource within a Kubernetes-cluster. In fact, a project has already been created to monitor the state of resources created by the cluster-api project: https://github.com/mercedes-benz/cluster-api-state-metrics

I hope this was useful, let's keep monitoring :)