CI/CD Automation with Jenkins, EKS, and Grafana in Kubernetes

In the ever-evolving world of software development, mastering the art of Continuous Integration and Continuous Deployment (CI/CD) has become indispensable. With the advent of containerization and orchestration platforms like Kubernetes, deploying and managing CI/CD pipelines has evolved into a sophisticated yet remarkably efficient process. This article takes you on a journey where we construct a real-world application-based project, commencing from the fundamentals of Amazon Elastic Kubernetes Service (EKS) and advancing into intricate subjects such as Jenkins, Prometheus monitoring, and more. So, fasten your seatbelts as we plunge into the realm of CI/CD in Kubernetes, all under the expert guidance of Vimal Daga Sir, our seasoned mentor who will illuminate the path ahead.

Within the pages of this comprehensive guide, we will delve into the deployment of Jenkins, EKS Kubernetes, and Grafana, crafting a powerful CI/CD toolchain that ushers in automation, scalability, and efficiency to our development processes. So, prepare for this exhilarating journey as we embrace the world of DevOps excellence, all under the guidance of a mentor who knows the terrain inside out and is eager to share invaluable wisdom.

Getting Started with EKS Kubernetes:

Our journey commences with the fundamentals of EKS Kubernetes, Amazon's managed Kubernetes service. We'll walk through the process of setting up an EKS cluster, configuring nodes, and getting our Kubernetes environment up and running. EKS serves as the robust foundation for our CI/CD pipeline, offering scalability, resilience, and ease of management.

Automating Jenkins CI/CD with Operators:

Jenkins is a powerhouse in the realm of CI/CD, and we'll harness its capabilities to streamline our development workflow. What's even more intriguing is our approach to deploying Jenkins using Kubernetes Operators. We'll delve into how Operators automate intricate application management tasks, simplifying the installation and configuration of Jenkins.

Managing Jenkins Pipelines as Code:

Modern CI/CD practices dictate that pipelines should be treated as code. We'll dive into Jenkins Pipeline as Code, leveraging Jenkins Kubernetes Custom Resource Definitions (CRDs) to define and manage our CI/CD pipelines. This approach ensures version control, collaboration, and the reproducibility of our pipelines.

Dynamic Pod Provisioning for Application Jobs:

In the dynamic world of Kubernetes, resource allocation and scaling are paramount. We'll learn how to configure application jobs to launch within dynamically provisioned pods in our Kubernetes deployment. This flexibility guarantees optimal resource utilization and scalability according to workload demands.

Monitoring Infrastructure Logs with Grafana Loki and Prom tail:

Effective monitoring plays a pivotal role in the world of CI/CD pipelines. We'll explore how Grafana Loki and Prom tail collaborate to assist us in collecting, storing, and querying log data from our Kubernetes infrastructure. This combination equips us with real-time insights into application performance and issues, greatly enhancing our ability to troubleshoot and optimize.

Metrics Monitoring with Prometheus:

To complete our monitoring stack, we'll integrate Prometheus, a leading open-source monitoring and alerting toolkit. Prometheus empowers us to collect and query performance metrics from both our Kubernetes cluster and applications. We'll establish custom alerts to proactively address issues, ensuring the reliability of our CI/CD pipeline.

Advanced Use Cases:

But our journey doesn't culminate here. We'll also explore advanced use cases, including creating an end-to-end pipeline using the operator, deploying pods with Helm charts, and delving into additional Kubernetes features to elevate our CI/CD process.

By the end of the blog, you'll have gained a comprehensive grasp of deploying Jenkins CI/CD on EKS Kubernetes, managing pipelines as code, dynamically provisioning pods, and establishing robust monitoring using Grafana Loki and Prometheus. You'll be equipped with the skills needed to thrive in the realm of CI/CD in Kubernetes, making you a valuable asset in the realm of modern software development. So, let's embark on this thrilling voyage into the heart of DevOps and Kubernetes.

Now, let's dive into the project!

Operators: The concept of Kubernetes' operator pattern allows us to extend the cluster's capabilities without altering Kubernetes' core code. Operators act as controllers for custom resources and interact with the Kubernetes API. In this blog, we embark on a journey to unlock the full potential of Kubernetes Operators by automating the deployment of Jenkins CI/CD and streamlining the setup of essential monitoring tools on an EKS cluster.

This task will involve the following key steps:

1. Tool Setup: Begin by installing essential tools, including AWS CLI, eksctl, kubectl, and Helm, to ensure you're well-equipped for the journey ahead.
2. Jenkins Operator Deployment: Utilize Helm to effortlessly deploy the Jenkins Operator on your EKS cluster, streamlining the process and ensuring effective management.
3. Jenkins Instance Configuration: Define the configuration of your Jenkins instance. This instance will be created and expertly managed by the Jenkins Operator within your EKS environment.
4. Dynamic Jenkins Agent Provisioning: Leverage the Jenkins instance to set up dynamic provisioning of Jenkins agents, optimizing resource allocation and scalability based on workload requirements.
5. Grafana Loki-Stack Deployment: Further enhance your project's capabilities by deploying the Grafana Loki-stack with the assistance of Helm. This stack is integral for monitoring infrastructure logs, system health, and performance.

1. Setting Up AWS EKS Cluster

1.1. Create an EKS cluster using eksctl:

To create an EKS cluster on us-east-1 region with default configurations using the following command:

$ eksctl create cluster --name devopscluster --region us-east-1        

1.2. Configure kubectl to connect to your EKS cluster:

After the cluster is successfully created and initialized, your kubeconfig file will be automatically updated. This file is essential for accessing your cluster. In case it's not updated automatically, you also have the option to update your kubeconfig file manually using the following AWS command. This ensures that you have the necessary access to your cluster for further operations.

$ aws eks --region us-east-1 update-kubeconfig --name devopscluster        

To check your nodes status use the following kubectl command.

$ kubectl get nodes -owide        

2. CI-CD using Jenkins-Operator on EKS

The Jenkins Operator is a Kubernetes Native Operator designed to handle operations for Jenkins on Kubernetes. It has been developed with a focus on immutability and a declarative Configuration as Code approach. This means it's designed to ensure stability and enable configuration through code-based definitions, making it a powerful tool for managing Jenkins within a Kubernetes environment.

2.1 Install Jenkins Operator using helm:

First, we will create a dedicated namespace for Jenkins where we will install our operator:

$ kubectl create namespace Jenkins         

Now let's install the operator by first adding the helm repository, then installing it using the following commands.

$ helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart        

$ helm install my-jenkins-operator jenkins/jenkins-operator -n jenkins --set jenkins.enabled=false        

We will not enable the Jenkins now which will deploy the Jenkins instance, we will deploy the Jenkins instance via YAML file in next step.

We can see that we have got a Jenkins resource in our Kubernetes cluster.

We won't activate Jenkins at this point, which would deploy the Jenkins instance. Instead, we'll deploy the Jenkins instance using a YAML file in the next step. As you can see, we already have a Jenkins resource in our Kubernetes cluster.

2.2 Launching Jenkins Instance:

By leveraging the Jenkins Custom Resource Definition (CRD) created by the Jenkins Operator, we can create a Jenkins resource within our Kubernetes cluster. This resource will be of kind "Jenkins," and its creation will initiate the deployment of a Jenkins instance.

The primary advantage of using the operator in this context is that it enables us to manage Jenkins in a Kubernetes-native manner. Below is the manifest file, 'jenkins_instance.yml,' which is used to create the Jenkins instance:

apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
  name: my-jenkins-instance
  namespace: jenkins
spec:
  service:
    type: LoadBalancer
    port: 8080
  configurationAsCode:
    configurations: []
    secret:
      name: ""
  groovyScripts:
    configurations: []
    secret:
      name: ""
  jenkinsAPISettings:
    authorizationStrategy: createUser
  master:
    disableCSRFProtection: false
    containers:
      - name: jenkins-master
        image: jenkins/jenkins:lts
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 12
          httpGet:
            path: /login
            port: http
            scheme: HTTP
          initialDelaySeconds: 100
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /login
            port: http
            scheme: HTTP
          initialDelaySeconds: 80
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 1500m
            memory: 3Gi
          requests:
            cpu: "1"
            memory: 500Mi
  seedJobs:
    - id: jenkins-operator
      targets: "cicd/jobs/*.jenkins"
      description: "Jenkins Operator repository"
      repositoryBranch: master
      repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git        

Within this Jenkins instance, we are specifying various configurations for the Jenkins instance. Here's a breakdown of some of these configurations:

Service: This configuration is used to create a load balancer resource for the Jenkins instance, allowing for load balancing and distribution of traffic.

Configuration As Code: It defines the configuration of Jenkins customization using the Configuration as Code Jenkins plugin. This is a powerful approach to managing Jenkins settings as code, ensuring consistency and reproducibility.

GroovyScripts: This part of the configuration defines customization of Jenkins using Groovy scripts. Groovy scripts are a flexible way to tailor Jenkins to specific requirements, and this configuration allows you to incorporate these customizations.

One main configuration from this file is seedJobs, here we can define our jobs that needs to be created by fetching the jenkins file from a remote repository where we store all our jenkins file.

$ kubectl create -f jenkins_instance.yml        

Once the manifest file is applied, we can see our jenkins-master instance is launched.

$ kubectl get pods -n jenkins        

Here we can see that my-jenkins-instance that is our jenkins-master is created, also we can see that a seed-job-agent pod is deployed that will configure our jobs in jenkins.

2.3 Accessing the Jenkins master instance.

To access the jenkins-master instance, we need to use the Loadbalancer service, and credentials to the user can be found in jenkins-operator-credentials-example.

$ kubectl get services -n jenkins
$ kubectl get secret jenkins-operator-credentials-example -n jenkins -o 'jsonpath={.data.user}' | base64 -d
$ kubectl get secret jenkins-operator-credentials-example -n jenkins -o 'jsonpath={.data.password}' | base64 -d        

As you can see our 2 jobs from the remote repository we gave are configured by the seed job that comes from the jenkins-operator.

2.4 Launch a job in jenkins to dynamically provision jenkins agent node.

Next, we'll initiate a job in Jenkins to dynamically provision a Jenkins agent node. Here are the steps to do this:

Begin by launching a dummy job to thoroughly test if our Jenkins instance is functioning correctly. Simply click on the job you want to run and select the "Build Now" option.

Check the console output to ensure that your Jenkins instance is operational and free from any issues.

After confirming the successful operation of your Jenkins instance, proceed to run the second job. This job is designed to dynamically provision a Jenkins agent node and execute your pipeline.

As the job progresses, you'll observe that the build was successful. In the process, you'll notice the console output, where Jenkins dynamically provisions a pod within the Kubernetes cluster to execute the job. The pod is terminated as soon as the job is completed.

This approach allows us to smoothly execute our Jenkins pipelines with the support of the Jenkins Operator on Kubernetes. It streamlines the process of managing Jenkins agents dynamically as needed for various tasks.

3. Grafana/Loki-stack for Monitoring Infrastructure Logs and Metrics

3.1 Install grafana/Loki-stack using Helm

First we will create a dedicated namespace called monitoring where we will install our operator:

$ kubectl create namespace monitoring        

Now lets install the operator by first adding the helm repository, then installing it using the following commands.

$ helm repo add grafana https://grafana.github.io/helm-charts        

$ helm upgrade --install loki grafana/loki-stack -n monitoring --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false        

Here, we're enabling Grafana and Prometheus. Currently, we don't require the creation of a persistent volume, so we have disabled that option. This streamlined process highlights the efficiency and convenience that operators bring to managing and integrating components, such as Prometheus, Grafana, and Loki, which would otherwise be time-consuming and complex if done manually.

3.2 Access the grafana GUI.

To access the grafana GUI first we have to expose the service, a clusterIP service is already created we will patch the service to make it as LoadBalancer type service.

$ kubectl patch svc loki-grafana -p '{"spec": {"type": "LoadBalancer"}}' -n monitoring        

To get the login-credentials we have can refer the secret created by the operator.

kubectl get secret loki-grafana -n monitoring -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'        

3.3 Monitoring logs on Grafana.

After login, go to explore tab, and select Loki, as the metrics database.

Now apply the following query to monitor our jenkins-instance.

{pod="jenkins-my-jenkins-instance"}        

Now let us try monitoring the memory usage, by our jenkins-master instance, these metrics are come from prometheus, so lets switch the DB for grafana as prometheus.

Now run the following promQL query to get the memory usage.

container_memory_usage_bytes{container="jenkins-master"}        

4. Cleanup: To delete the ekscluster:

eksctl delete cluster -n devopscluster        

Conclusion

In this journey exploring the realm of Kubernetes Operators, we've experienced their remarkable capabilities for automating and simplifying CI/CD workflows on Amazon EKS. By leveraging Kubernetes Operators, we've accomplished the following:

1. Efficient CI/CD Automation: Kubernetes Operators have made the deployment of Jenkins CI/CD pipelines on EKS a seamless and automated process. The Operator pattern allowed us to enhance the cluster's capabilities without the need to modify Kubernetes' core code.

2. Dynamic Resource Provisioning: With the help of Kubernetes Operators, we've enabled dynamic provisioning of Jenkins agents, resulting in optimized resource allocation and scalability.

3. Streamlined Monitoring: Through the utilization of Grafana Loki, Promtail, and Prometheus, we've effortlessly monitored infrastructure logs and collected valuable metrics. This has empowered us to make informed decisions and maintain a healthy and performant environment.

As we wrap up this exciting journey through the world of CI/CD with EKS Kubernetes, it's a good time to reflect on the ground we've covered. We've gone from the basics of setting up EKS Kubernetes to the intricacies of automating Jenkins CI/CD using Operators, managing pipelines as code, and implementing robust monitoring with Grafana Loki and Prometheus.

But let's not just focus on the technical skills and knowledge you've gained; let's also appreciate the journey itself. Each step you've taken and every challenge you've overcome have brought you closer to becoming a master of CI/CD within the Kubernetes universe.

I want to express my sincere gratitude to you, the reader, for embarking on this adventure with me. Your curiosity, dedication, and thirst for knowledge have been the driving forces behind your achievements. Remember, the skills you've acquired here are not just tools in your toolbox; they are the keys that unlock new possibilities in your career. Keep pushing boundaries, keep automating, and keep reaching for new heights.