Integrating Prometheus & Grafana using Kubernetes.

Task in Hand :

Integrate Prometheus and Grafana and perform in following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services.
2. make their data to be remain persistent .
3. both of them should be exposed to outside world.

Pre-requisites:

1. Minikube installed and running
2. kubectl configured and added to the path

ConfigMap :

ConfigMap is used to store the configurations of the application running in Pod. For example, the httpd.conf for httpd. You could modify the httpd.conf file in Pod but everything will be gone once Pod is replace. So we can write the desired configurations in ConfigMap and tell Pod to use such configuration whenever Pod is replaced. Therefore Pod is always running with how we wanted it to be.

PVC :

when implementing Prometheus and Grafana on K8S. Everything will work fine but as we know that Pod is stateless and could be recycle anytime, therefore any settings configured on Prometheus and Grafana would be gone when Pod is replaced.

Therefore, to solve this problem, we need to use Persistent Volumes(PV), PersistentVolumeClaims(PVC) and ConfigMap.

PVs as named the provisioned storage space in cluster to store data. Data is stored in PV rather than Pod itself, so data is retain when Pod is replaced. PVC as named is used to claim the provisioned storage space for use.

Service :

This is used to expose the pods services with a port number so that it can be accessible from the outside world.

The default port range for kubernetes services is in between 30000-32767, you can select any between them or kubernetes will provide automatically any random port in that range.

Manifest file for Prometheus :

Prometheus ConfigMap :

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |-
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:

        - targets: ['localhost:9090']

Prometheus PVC :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pvc
  labels:
    app: prometheus
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:

      storage: 20Gi

Prometheus Service :

we will be able to access the Prometheus WebUI at https://<minikube ip>:30001 .

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  ports:
    - port: 9090
      nodePort: 30001
  selector:
    app: prometheus
    tier: backend

  type: NodePort

Prometheus Deployment :

Now we have to create the deployment for prometheus. It will pull the prometheus image from docker hub, configure it, then create and mount the persistent volume at the destination folder.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      tier: backend
  strategy:
    type: Recreate
  template:
    metadata:
      labels: 
        app: prometheus
        tier: backend
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus
        - name: data
          mountPath: /prometheus
        ports:
        - containerPort: 80
      securityContext:
        runAsUser: 0
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config
        - name: data
          persistentVolumeClaim:

            claimName: prometheus-pvc

Manifest file for Grafana :

Grafana PVC :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  labels:
    app: grafana
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:

      storage: 20Gi

Grafana Service :

we will be able to access the Grafana WebUI at https://<minikube ip>:30002 .

apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  ports:
    - port: 3000
      nodePort: 30002
  selector:
    app: grafana
    tier: frontend

  type: NodePort

Grafana Deployment :

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: grafana
  labels:
    app: grafana
    tier: frontend
spec:
  selector:
    matchLabels:
      app: grafana
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: grafana
        tier: frontend
    spec:
      containers:
      - image: grafana/grafana:latest
        name: grafana
        ports:
        - containerPort: 3000
          name: grafana
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:

          claimName: grafana-pvc

Note : While merging the all the parts of Prometheus in one YAML file and all the parts of Grafana into other YAML file, you need to seperate all the parts with " --- " as shown below, this tells yaml that they are 2 separate parts.

Kustomization.yaml :

Now i'm making a separate Kustomization.yaml file which will bind the two resources (prometheus-deployment.yaml and grafana-deployment.yaml).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - prometheus-deployment.yaml

  - grafana-deployment.yaml

now you can deploy the complete setup by just running one command...

kubectl create -k .

if you run this command, you will see all services will be created automatically.

to view the deployments...

kubectl get deployments

to view the ConfigMaps...

kubectl get cm

to view the pods...

kubectl get pods

to view the PVCs...

kubectl get pvc

to view the service..

kubectl get svc

to view the replica sets...

kubectl get replicasets

or to view all at once you can use...

kubectl get all

Now to view the Prometheus and Grafana WebUI use the minikube ip and the mentioned port number.

to know your minikube ip...

minikube ip

Prometheus WebUI :

to acces the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.

https://<minikube ip>:30001

and this will be the first page you will see

enter a query in the Expression text field and click the execute button to get the value or click on graph to get a graph for visualization.

for graph

now go to the Status > Targets, then you will see the target from where the metrics are obtained.

Grafana WebUI :

to acces the WebUI you need to go to the minikube ip with the provided port as mentioned earlier.

https://<minikube ip>:30002

when logged in for the first time the user and password will be admin, then after you can change the password of it.

then this will be the Dashboard you will see after you have logged in to the Grafana.

click on the Data Sources to add your data sources.

select prometheus as your data source.

enter a name and url of the prometheus server in the given fields.

then click on save and test, you will get a prompt saying datasource added if there isn't any kind of error.

then go to the " Home > Create > Dashboard > Add Panel " to create a new dashboard.ad execute the same two queries you executed in the prometheus you will get the same graphs.

you can change the time, type of graphs and many more using graphana.

save the panel, if you see carefully it also display the query and the colour of the graphs for each query when the cursor is moved on the graph.

you can add as many panel as you want with different queries.

as the pods have a persistent storage, even the pods crashes as your data is saved in a PVC your data will not be lost and replica set will launch a new pod and connect it with the PVC instantly.