Integrating Prometheus & Grafana on Top of Kubernetes

What is Kubernetes ?

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. With Kubernetes we can:

• Orchestrate containers across multiple hosts.
• Make better use of hardware to maximize resources needed to run your enterprise apps.
• Control and automate application deployments and updates.
• Mount and add storage to run stateful apps.
• Scale containerized applications and their resources on the fly.
• Declaratively manage services, which guarantees the deployed applications are always running the way you intended them to run.
• Health-check and self-heal your apps with autoplacement, autorestart, autoreplication, and autoscaling.

What is Prometheus ?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company.

What is Grafana ?

Grafana is open source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Problem Statement :

Integrate Prometheus and Grafana and perform in following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pods or Services.
2. And make their data to be remain persistent.
3. And both of them should be exposed to outside world.

That is our setup will be something like this :

Task Description :

1. Launching Prometheus on K8s :

YAML Code for ConfigMap for Prometheus :

• ConfigMap is used to pass configuration file to any pod which is copied to a volume, making configuration file persistent.
• Here we have configured Prometheus to scrape metrics for its own Prometheus service.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prom-configfile
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    alerting:
      alertmanagers:
      -  static_configs:
         -  targets:
    rule_files:
    scrape_configs:
      -  job_name: 'prometheus'
         static_configs:
         
         -  targets: ['localhost:9090']

YAML Code for generating Persistent Volume for Prometheus :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: prom-pv-claim
    labels:
        app: prom
        env: prod
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

YAML Code for Creating Deployment for Prometheus :

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: prom
  labels:
    app: prom
spec:
  selector:
    matchLabels:
      app: prom
      env: prod
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: prom
        env: prod
    spec: 
      containers:
      - image: prom/prometheus
        name: prom
        args:
        - '--config.file=/prometheus.yml'
        - '--storage.tsdb.path=prometheus_data/'
        ports:
        - containerPort: 9090
          name: prom
        volumeMounts:
        - name: prometheus-persistent-storage
          mountPath: prometheus_data/
        - name: prometheus-configfile
          mountPath: /prometheus.yml
          subPath: prometheus.yml
      volumes:
      - name: prometheus-persistent-storage
        persistentVolumeClaim:
          claimName: prom-pv-claim
      - name: prometheus-configfile
        configMap:
          name: prom-configfile
          defaultMode: 0744

YAML Code for Exposing Prometheus Deployment Through Service :

apiVersion: v1
kind: Service
metadata:
  name: prom
  labels:
    app: prom
spec:
  selector:
    app: prom
    env: prod
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30000

Kustomization File :

• kustomization.yml file helps us to creates multiple Kubernetes Service in one single command
• -k option in kubectl create automatically creates the kustomization.yml file in the current directory.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization


resources:
  - pvc-prom.yml
  - configmap-prom.yml
  - deploy-prom.yml
  - svc-prom.yml
  

> kubectl create -k .

Prometheus Launched :

2. Launching Grafana on Kubernetes :

YAML Code for generating ConfigMap for Grafana :

• Here ConfigMap is used to make Data Source Of Grafana Persistent.

kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-config
data:
  datasource.yaml: |
    apiVersion: 1	
    datasources:
      - name: Prometheus
        type: prometheus
        access: server
        url: https://192.168.99.108:30000
        version: 1
        editable: true

YAML Code for generating Persistent Volume for Grafana :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: grafana-pv-claim
    labels:
        app: grafana
        env: prod
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

YAML Code for Creating Deployment for Grafana:

apiVersion: apps/v1 
	kind: Deployment
	metadata:
	  name: grafana
	  labels:
	    app: grafana
	spec:
	  selector:
	    matchLabels:
	      app: grafana
	      env: prod
	  strategy:
	    type: Recreate
	  template:
	    metadata:
	      labels:
	        app: grafana
	        env: prod
	    spec:
	      containers:
	      - image: grafana/grafana
	        name: grafana
	        ports:
	        - containerPort: 3000
	          name: grafana
	        volumeMounts:
	        - name: grafana-persistent-storage
	          mountPath: /var/lib/grafana/
	        - name: grafana-script
	          mountPath: /etc/grafana/provisioning/datasources/datasource.yaml
	          subPath: datasource.yaml
	      volumes:
	      - name: grafana-persistent-storage
	        persistentVolumeClaim:
	          claimName: grafana-pv-claim
	      - name: grafana-script
	        configMap:
	          name: grafana-config

?YAML Code for Exposing Grafana Deployment Through Service :

apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  selector:
    app: grafana
    env: prod
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      
      nodePort: 30001

Kustomization File :

• kustomization.yml file helps us to creates multiple Kubernetes Service in one single command
• -k option in kubectl create automatically creates the kustomization.yml file in the current directory.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization


resources:
  - pvc-graf.yml
  - deploy-graf.yml
  - svc-graf.yml
  - configmap-graf.yml

> kubectl create -k .

Grafana Launched :

Finally we have integrated Prometheus as the Data Source of Grafana and created beautiful graph for monitoring :

Github Repository

Thank you !!