Prometheus and Grafana on the top of Kubernetes

We are working in a Kubernetes (k8s) environment, then we proceeded to choose a monitoring solution. We decided to deploy Prometheus and Grafana on the k8s Cluster.

Grafana is an “open platform for beautiful analytics and monitoring”. It will visualize your collected metrics and present them in a graphical way (line charts, histograms, gauges, tables, etc.) As an open-source application, you can deploy it in many ways I choose simplest one like First build docker Image of Prometheus and Grafana then send to docker hub and pull from there.

Prometheus is also a free software application used for event monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting

In our case, we Integrate both With Kubernetes.

So Lets Begin,

Now, if we deploy these tools on a container orchestration engine like Kubernetes, here, we face persistent or we can say permanent storage issue because by default they take ephemeral storage. Means, as Kubernetes works on the concept of pods, and as soon as the pod fails due to any reason, the whole data stored corresponding to that pod will also be deleted or lost. This problem can be resolved by using a Kubernetes resource named Persistent Volume (PV).

PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. A PersistentVolumeClaim (PVC) is a request for storage by a user.

PV can be considered as external storage which can be mount on the pod and would not be affected even if the pods get deleted, hence making our data persistent and safe!!

Here we will see how can we use PVC to have a Persistent storage.

Step 1:

let's create a PVC template file, which will create a PVC to be further used for our Prometheus and Grafana pods. So, here I a creating two file one for Prometheus and second one for Grafana.

prome_pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: prom-pvc
    labels:
        app: prometheus
spec:
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 2Gi

grafana_pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: graf-pvc
    labels:
        app: grafana
spec:
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 2Gi

To run the above file so as to create the PVC, use the below commands:

> kubectl create -f prometheus_pvc.yml

> kubectl create -f grafana_pvc.yml

Now, we just need to launch the pods for Grafana and Grometheus. While deploying those pods over K8s, we need to simply specify the PVC name that we have created using the above files in order to mount the storage on pods.

We will mount the PVC over the file where we want our data not to be lost. In the case of Prometheus, the path would be /etc/prometheus/data. And similarly, in case of grafana, it would be /var/lib/grafana.

We can write the following code to deploy the pods along with persistent storage:

prome_dep.yml

apiVersion: apps/v1
kind: Deployment
metadata:
    name: prom-dep
    labels:
        app: prometheus


spec:
    selector:
        matchLabels:
            app: prometheus
    strategy:
        type: Recreate
    template:
        metadata:
            labels:
                app: prometheus
        spec:
            containers:
            - image: prom/prometheus:latest
              name: prometheus
              volumeMounts:
              - name: prom-vol
                mountPath: /etc/prometheus/data
            volumes:
            - name: prom-vol
              persistentVolumeClaim:
                  claimName: prom-pvc

grafana_dep.yml

apiVersion: apps/v1
kind: Deployment
metadata:
    name: graf-dep
    labels:
        app: grafana


spec:
    selector:
        matchLabels:
            app: grafana
    strategy:
        type: Recreate
    template:
        metadata:
            labels:
                app: grafana
        spec:
            containers:
            - image: grafana/grafana:latest
              name: grafana
              volumeMounts:
              - name: graf-vol
                mountPath: /var/lib/grafana
            volumes:
            - name: graf-vol
              persistentVolumeClaim:
                  claimName: graf-pvc

Execute the files to deploy the pods :

> kubectl create -f prometheus_dep.yml

> kubectl create -f grafana_dep.yml

We have mounted a PVC over the pods achieving persistent storage.

To use them, we need to expose the pods on some port number as shown below:

> kubectl expose deployment prom-dep --port=9090 --type=NodePort

> kubectl expose deployment graf-dep --port=3000 --type=NodePort

We can check the details using command : it shows wide details about pods and deployments.

> kubectl get all

In the Prometheus by default work on port 9090 is bound to 31209, and Grafana port 3000 is bound to 31058. So we can access them using the given ports.

Now open the browser and type your IP:31209 you can see like this. Here you can do anything about the fetching CPU metrics, ram utilization, run prome queries anything.

Now Open Grafana dashboard by typing the IP: 31058 in browser. you will be able to see like this and start monitoring and create new new dashboard for monitoring cpu utilization, ram metrics, nodes health and many more.

That's all

Hope You Liked It!!

if any query feel free DM me.