Integrate Prometheus and Grafana over Kubernetes Cluster

Prometheus:

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud.

Prometheus's main features are:

• A multi-dimensional data model with time series data identified by metric name and key/value pairs
• PromQL, a flexible query language to leverage this dimensionality
• No reliance on distributed storage; single server nodes are autonomous
• Targets are discovered via service discovery or static configuration
• Multiple modes of graphing and dashboarding support

Grafana:

Grafana is multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is expandable through a plug-in system. End users can create complex monitoring dashboards using interactive query builders.

Grafana connects with every possible data source, commonly referred to as databases such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL, etc.

Grafana being an open-source solution also enables us to write plugins from scratch for integration with several different data sources.

The tool helps us study, analyze & monitor data over a period of time, technically called time-series analytics.

If you want to see the working of Graphana click here.

Task Description:

Integrate Prometheus and Grafana and perform in the following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services.

2. And make their data to remain persistent.

3. And both of them should be exposed to the outside world.

Steps to achievement:

Step 1: We make separate images of Prometheus and Grafana on docker.

Docker file for Prometheus:

FROM centos


RUN yum install wget -y


RUN wget https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz


RUN tar -xzf prometheus-2.19.0.linux-amd64.tar.gz


RUN mkdir /prometheus

CMD ./prometheus-2.19.0.linux-amd64/prometheus --config.file=/prometheus-2.19.0.linux-amd64/prometheus.yml --storage.tsdb.path=/prometheus && tail -f /dev/null

Dockerfile used to build Grafana image:

FROM centos


RUN yum install wget -y


RUN wget https://dl.grafana.com/oss/release/grafana-7.0.3-1.x86_64.rpm


RUN yum install grafana-7.0.3-1.x86_64.rpm -y


WORKDIR /usr/share/grafana

CMD /usr/sbin/grafana-server start && /usr/sbin/grafana-server enable && /bin/bash

Command to build an image is:

docker build -t image_name:version /path

After building you can also push the image on docker hub using the following command:

docker push image_name

• Step 2: Creating a Persistent Volume on Kubernetes that will be used for storing data for both Prometheus and Grafana using YAML.

Code to create a PV for Grafana:

apiVersion: v1


kind: PersistentVolume


metadata:


   name: grafana-pv
   labels:
     type: local


spec:
   storageClassName: manual
   capacity:
     storage: 2Gi
   accessModes:
     - ReadWriteOnce
   hostPath: 
     path: "/mnt/sda1/data/grafana"

The host path is the location of the folder for which we want to make persistent that will be mounted on the pods whenever it is created. It can be where ever you want it to be on the local system.

Code to create a PV for Prometheus:

apiVersion: v1


kind: PersistentVolume


metadata:


   name: prom-pv
   labels:
     type: local


spec:
   storageClassName: manual
   capacity:
     storage: 2Gi
   accessModes:
     - ReadWriteOnce
   hostPath:
     path: "/mnt/sda1/data/prometheus" 

Step 2: We create Persistent Volume Claims for both Prometheus and Grafana.

Code to create a PVC (Persistent Volume Claim) for Grafana:

apiVersion: v1
kind: PersistentVolumeClaim

metadata:
   name: grafana-pvc
   labels:
     name: grafana-pvc

spec:
   storageClassName: manual
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 2Gi

Code to create a PVC for Prometheus:

apiVersion: v1
kind: PersistentVolumeClaim

metadata:
   name: prom-pvc
   labels:
     name: prom-pvc

spec:
   storageClassName: manual
   accessModes:
     - ReadWriteOnce
   resources:
     requests:

       storage: 2Gi

Step 3: Creating a deployment for both Prometheus and Grafana that will create the pods and automatically attach the respective PV.

Deployment for Grafana:

apiVersion: apps/vi 
kind: Deployment


metadata:
   name: grafana-deploy


spec:
   replicas: 1
   selector:
     matchLabels:
       app: grafana


   template:
     metadata:
       name: grafana-deploy
       labels:
         app: grafana


     spec:
       volumes:
       - name: grafana-storage
         persistentVolumeClaim:
           claimName: grafana-pvc
       containers:
         - name: grafana
           image: asd/grafana:v1
           imagePullPolicy: IfNotPresent 
           volumeMounts:
             - mountPath: "/data-grafana"
               name: grafana-storage
       restartPolicy: Always
         

Deployment for Prometheus:

apiVersion: apps/vi 
kind: Deployment


metadata:
   name: prom-deploy


spec:
   replicas: 1
   selector:
     matchLabels:
       app: prometheus


   template:
     metadata:
       name: prom-deploy
       labels:
         app: prometheus


     spec:
       volumes:
       - name: prom-storage
         persistentVolumeClaim:
           claimName: prom-pvc
       containers:
         - name: prom
           image: asd/prometheus:v1
           imagePullPolicy: IfNotPresent 
           volumeMounts:
             - mountPath: "/data-prometheus"
               name: prom-storage
       

At last, we have to make a kustomization.yml file. It is used to bind all the files together and to run them in a particular order so there are no bottlenecks.

apiversion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - grafana-pv.yml
  - prom-pv.yml
  - grafana-pvc.yml
  - prom-pvc.yml
  - grafana-deploy.yml
  - prom-deploy.yml

Command to execute the kutomization.yml file is:

kubectl create -k .

Command to expose the ports are:

kubectl expose deployment prom-deploy --port=9090 --type=NodePort

kubectl expose deployment grafana-deploy --port=3000 --type=NodePort

These are the default ports used by Prometheus and Grafana respectively.

After exposing use the following command to check if everything is up and running perfectly.

kubectl get all

Now you can go over to the Prometheus and Grafana IP on the exposed ports to see the output as given below.

We are all done with are setup.

Thank you!