DevOps AL Task5-PROMETHEUS AND GRAFANA WITH PERSISTENT STORAGE USING KUBERNETES

PROBLEM STATEMENT:

Integrate Prometheus and Grafana and perform in the following way:

1. Deploy them as pods on top of Kubernetes by creating resources Deployment, ReplicaSet, Pod or Service

2. And make their data to remain persistent 

3. And both of them should be exposed to the outside world

EXPLANATION:

MINIKUBE: Minikube is a lightweight version of Kubernetes which is much easier to use locally. It will create a VM on your local machine where you can run a single-node cluster. This is useful for testing

CONTAINERS: A container is a standardized structure that can store and run any application.

PODS: A pod holds one or more container(s). Pods are the simplest unit that exists within Kubernetes.

SERVICES: A service is an API object that exposes an application, it essentially describes how network traffic should access a set of pods. Services are found on every node.

DEPLOYMENT: Deployments are an API object, they essentially manage pod replication. A deployment defines the state of your cluster – for example, how many replicas of a pod should be running. When the deployment is added to a cluster, Kubernetes will automatically make the correct number of pods and then monitor them. If a pod fails, Kubernetes will replicate it, following the ‘deployment’ criteria.

PROMETHEUS?

Prometheus is a leading open-source metric instrumentation, collection, and storage toolkit built at SoundCloud beginning in 2012. Since then, it's graduated from the Cloud Native Computing Foundation and become the de facto standard for Kubernetes monitoring.

GRAFANA?

Grafana is an open-source platform for data visualization, monitoring, and analysis. Used by thousands of companies to monitor everything from infrastructure, applications, and power plants to beehives

PROMETHEUS+GRAFANA:

Grafana supports querying Prometheus. The Grafana data source for Prometheus is included since Grafana 2.5.0 (2015-10-28). By default, Grafana will be listening to https://localhost:3000. The default login is "admin" / "admin".

STEPS:

STEP 1) Setup Kubernetes and minikube in your base OS. And start the Kubernetes service:

minikube start

After the successful start of Kubernetes. Run the following command to check for any resource running, if running then clean the workspace(kubectl delete all --all).

STEP 2) To setup Prometheus and Grafana I used two images available at Docker Hub(public registry/repository). These are official Prometheus and Grafana images available. One can also create their own image and push it onto Docker and use it.

Created three YML files grafpvcDeploy.yml,grafpvcservice.yml and kustomization.yml.The grafpvcDeploy.yml file creates PVC for both Grafana and Prometheus respectively and creates Deployment consisting of two pods-Grfana and Prometheus with their respective PVC's attached and mounted to keep data persistent in case of pod failure.

grafpvcDeploy.yml:

This file as you can see creates a PVC for Grafana (graf-pvc) and also pvc for Prometheus individually with ReadWriteOnce Access mode and size 2Gib. The three dashes separate the parts of code for launching each resource. The Pvc dynamically creates PV as well.

Then the deployment code creates a deployment consisting of two pods-Prometheus and Grafana. There respective PVC's are attached and mounted and defined a port-9090 for Prometheus & 3000 for Grafana.

grafpvcservice.yml:

This launches a service. By assigning ports such that every time pod restarts after failure or deletion , they are assigned the same port numbers.

kustomization.yml:

Kustomize is a tool that lets you create an entire Kubernetes application out of individual pieces — without touching the YAML for the individual components. This file here will automatically scrape out the specified two yml files from the same folder and launch it. All in one click. How awesome is that!

run the kustomization file:

As it can be seen, the resources have been successfully created, with no error.

STEP 3) Get the minikube IP:

Launch the Prometheus and Grafana WebUI using respective port no's:

This is what it will look like.

Creating a Prometheus data source:

1. Click on the "cogwheel" in the sidebar to open the Configuration menu.
2. Click on "Data Sources".
3. Click on "Add data source".
4. Select "Prometheus" as the type.
5. Set the appropriate Prometheus server URL (for example, https://localhost:9090/), here its https://192.168.99.104:30001
6. Adjust other data source settings as desired (for example, choosing the right Access method).
7. Click "Save & Test" to save the new data source.

Creating a Prometheus graph:

1. Click the graph title, then click "Edit".
2. Under the "Metrics" tab, select your Prometheus data source (bottom right).
3. Enter any Prometheus expression into the "Query" field, while using the "Metric" field to lookup metrics via autocomplete.

4. Tune other graph settings until you have a working graph.

STEP 4) Now to test if the data remains persistent .Delete the setup:

On deletion, the server won't be responsive now.

But here,pvc is still intact, since they weren't deleted. Now if you run kubectl create command again for kustomization.yml file the whole setup will be launched again with the same pvc attached and keeping data intact.

That's it! Our data remains safe even after deleting and restarting the setup.

Thank you.

Github repository link for following YML files:https://github.com/TanyaChetnaVaish/devaltask5