EKS Kubernetes to deploy Jenkins CI/CD tool using Automated way Operator Monitoring of Infrastructure log by Grafana Loki & Promtail

EKS Kubernetes to deploy Jenkins CI/CD tool using Automated way Operator, and manage Jenkins Pipeline as Code by Jenkins Kubernetes CRD and Application jobs launches in dynamic provisioning node our kubernetes deployment pods and monitoring of infrastructure log by Grafana Loki and Promtail and Metrics by Prometheus Monitoring Tool.

Introduction

In the rapidly evolving domain of software development, streamlining efficient CI/CD pipelines is of paramount importance. Amazon Elastic Kubernetes Service (EKS), a managed Kubernetes solution on AWS, plays a pivotal role in this landscape. This article delves into the process of deploying Jenkins for CI/CD on EKS Kubernetes by leveraging automated operators for a seamless setup. Beyond deployment, we explore the benefits of managing Jenkins pipelines as code through Jenkins Kubernetes Custom Resource Definitions (CRDs). This approach facilitates the automatic initiation of application tasks on dynamically provisioned nodes, optimizing the deployment of pods within our Kubernetes infrastructure. But the journey doesn't stop there—our exploration extends to the critical realm of monitoring, where Grafana Loki and Promtail handle infrastructure logs, and Prometheus oversees metrics. This comprehensive ecosystem promises to revolutionize your software development, enhancing precision, scalability, and providing insights into both application performance and infrastructure health. All under the expert guidance of Mr. Vimal Daga Sir. The convergence of EKS Kubernetes, Jenkins, and robust monitoring tools creates a data-driven CI/CD environment that is robust and efficient.

Getting Started with EKS Kubernetes: Our expedition commences with the foundational knowledge of EKS Kubernetes, Amazon's managed Kubernetes service. We'll uncover the steps for creating an EKS cluster, configuring nodes, and getting our Kubernetes environment up and operational. EKS serves as the bedrock for our CI/CD pipeline, providing scalability, resilience, and ease of management.

Optimizing Jenkins CI/CD with Operators: Jenkins is a powerhouse in the world of CI/CD, and we'll harness its capabilities to streamline our development processes. What makes this even more exciting is the deployment of Jenkins through Kubernetes Operators. We'll delve into how Operators automate intricate application management tasks, simplifying the installation and configuration of Jenkins.

Managing Jenkins Pipelines as Code: Modern CI/CD practices dictate that pipelines should be treated as code. We'll explore Jenkins Pipeline as Code, utilizing Jenkins Kubernetes Custom Resource Definitions (CRDs) to define and oversee our CI/CD pipelines. This approach ensures version control, collaborative work, and the ability to replicate our pipelines.

Dynamic Pod Provisioning for Application Tasks: In the Kubernetes landscape, resource allocation and scaling are dynamic endeavors. We'll learn how to configure application tasks to launch within dynamically provisioned pods in our Kubernetes setup. This flexibility ensures efficient resource utilization and scalability in response to workload variations.

Monitoring Infrastructure Logs with Grafana Loki and Promtail: Effective monitoring is imperative for CI/CD pipelines. We'll delve into how Grafana Loki and Promtail work together to assist us in collecting, storing, and querying log data from our Kubernetes infrastructure. This combination provides real-time insights into application performance and issues, thereby enhancing our ability to troubleshoot and optimize.

Metrics Monitoring with Prometheus: To round out our monitoring framework, we'll integrate Prometheus, a leading open-source monitoring and alerting system. Prometheus empowers us to collect and query performance metrics from our Kubernetes cluster and applications. We'll configure custom alerts to proactively address issues, ensuring the reliability of our CI/CD pipeline.

Let's Start

1. AWS Cloud Configuration:

• Establish an AWS account if you don't already possess one.
• Generate an IAM user with the necessary permissions, such as AdministratorAccess.
• Record the access key and secret key associated with the user.

2. AWS CLI Setup:

• Establish the AWS Command Line Interface (CLI) by obtaining it through downloading.

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html?source=post_page-----7f62cdecf1cd--------------------------------

• Execute the aws configure command and supply the access key, secret key, region, and preferred output format when prompted.

3. Install EKSctl:

• EKSctl is a command-line utility for the creation and administration of Amazon EKS clusters.
• Download EKSctl from the [GitHub releases] page.

https://github.com/eksctl-io/eksctl/releases?source=post_page-----7f62cdecf1cd--------------------------------

• Install EKSctl by placing the binary in a directory included in your path.

4. Create an Amazon EKS Cluster:

• Use EKSctl to create an EKS cluster, e.g.

$ eksctl create cluster - name mycluster1 - region us-west-2        

• Verify the cluster creation with

$ eksctl get cluster - name mycluster1 - region us-west-2        

• Ensure you have kubectl installed to interact with the cluster. You can follow the installation steps [here]

https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/

5. Helm Installation:

• Download Helm from the [Helm releases] page and install it.

https://github.com/helm/helm/releases

6. Install Jenkins Operator:

$ kubectl create namespace lwns        

• Now add the Jenkins Helm repository

$ helm repo add jenkins
https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart        

• Install the Jenkins Operator with custom settings

$ helm install my-jenkins-operator jenkins/jenkins-operator -n lwns - set jenkins.enabled=false         

• Monitor the operator pods with

$ kubectl - namespace lwns get pods -w        

7. Create Jenkins Instance:

Create a YAML file

$ nano jenkins_instance.yaml        

Here is the manifest file jenkins_instance.yml for creating the Jenkins instance

apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
  name: example
  namespace: lwns
spec:
  configurationAsCode:
    configurations: []
    secret:
      name: ""
  groovyScripts:
    configurations: []
    secret:
      name: ""
  jenkinsAPISettings:
    authorizationStrategy: createUser
  master:
    disableCSRFProtection: false
    containers:
      - name: jenkins-master
        image: jenkins/jenkins:lts
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 12
          httpGet:
            path: /login
            port: http
            scheme: HTTP
          initialDelaySeconds: 100
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /login
            port: http
            scheme: HTTP
          initialDelaySeconds: 80
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 1500m
            memory: 3Gi
          requests:
            cpu: "1"
            memory: 500Mi
  seedJobs:
    - id: jenkins-operator
      targets: "cicd/jobs/*.jenkins"
      description: "LW Jenkins Operator repository"
      repositoryBranch: master
      repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git        

• Apply the configuration to initiate the creation of the Jenkins instance.

$ kubectl create -f jenkins_instance.yaml        

• Monitor the Jenkins instance deployment with

$ kubectl -namespace lwns get pods -w        

8. Accessing the Jenkins:

• Jenkins credentials

$ kubectl - namespace lwns get secret jenkins-operator-credentials-example -o 'jsonpath={.data.user}' | base64 -d
kubectl - namespace lwns get secret jenkins-operator-credentials-example -o 'jsonpath={.data.password}' | base64 -d        

• Begin port-forwarding to access Jenkins locally

$ kubectl - namespace lwns port-forward jenkins-example 7070:7070        

9. Install Grafana Loki and Prometheus:

• Grafana Helm repository

$ helm repo add grafana https://grafana.github.io/helm-charts        

• First, add the Helm repository, and then execute the given commands to install the operator.

$ helm upgrade --install loki grafana/loki-stack -n monitoring --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false        

• Set up Grafana as a LoadBalancer service

$ kubectl patch svc loki-grafana -p '{"spec": {"type": "LoadBalancer"}}'        

10. Access Grafana and Configure Loki setup:

Sign in to Grafana using the provided hostname and configure data sources and dashboards. Configure Loki as a data source in Grafana.

• The hostname of the load balancer:-

$ kubectl get svc loki-grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'        

• Visit the link provided by amazonaws.com and log in to Grafana using your username and password.
• Wait for approximately 2 minutes while AWS LoadBalancer completes its setup.
• Afterward, execute this command to obtain your Grafana username and password:

$ kubectl get secret loki-grafana -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'        

• Monitoring logs on Grafana After login, go to explore tab, and select Loki, as the metrics database.

Now infrastructure setup is now complete and ready to roll. This comprehensive guide equips you to establish Jenkins CI/CD using Jenkins Pipeline as Code, implement Grafana Loki for log monitoring, and employ Prometheus for metrics tracking on Amazon EKS. Be sure to tailor the configurations to align with your unique requirements and use cases. With these tools and practices in place, you're well-prepared to streamline your development processes, enhance monitoring capabilities, and boost the efficiency and reliability of your software deployment workflows. This setup ensures that you're well-equipped to adapt and optimize your CI/CD environment as your project evolves.

CLEANUP: To delete the ekscluster

$ eksctl delete cluster -n devopscluster        

As we conclude this exciting journey through the realm of CI/CD with EKS Kubernetes, it's essential to pause and reflect. We've traveled from the fundamentals of setting up EKS Kubernetes to automating Jenkins CI/CD through Operators, managing pipelines as code, and establishing robust monitoring with Grafana Loki and Prometheus.

However, beyond the technical expertise gained, it's crucial to acknowledge the journey itself. Every step taken, every obstacle surmounted, has brought you closer to mastering CI/CD within the Kubernetes environment.

I extend my sincere gratitude to you, the reader, for embarking on this adventure. Your curiosity, commitment, and hunger for knowledge are the driving forces behind your achievements. Remember that the skills acquired here are not just tools but the means to explore new horizons in your career.

As you continue your CI/CD journey, keep exploring, experimenting, and learning. The world of DevOps and Kubernetes is ever-changing, and your expertise is the compass guiding you through its complexities.

Here's to your ongoing success, your insatiable thirst for knowledge, and your unwavering commitment to mastering the art of CI/CD. Keep pushing boundaries, automating, and reaching new heights. The future is yours to conquer!

THANK YOU!