Creating a CI/CD pipeline using Groovy
This small project focuses on how to use scripts in Groovy Language used as DSL (Domain-specific Language) for Jenkins. Using groovy, a pipeline can be created similar to the one created in this article, but this time with complete automation.
Part I : using Seed Job
- Create container image that’s has Jenkins installed using Dockerfile. When we launch a container using this image, it should automatically start Jenkins service.
- Create a Seed Job to pull the GitHub repository automatically when some developers push/updates new codes to some GitHub repository.
- The seed job must run a script in Groovy Language written by the developer. This creates a job chain of 4 jobs using build pipeline plugin in Jenkins.
Part II : deploying Website
The pipeline and 4 jobs generated as according to the Groovy script, will be used to contact a Kubernetes Cluster to launch a webpage as a deployment with a persistent volume and a fourth job to mail the developer if the webpage fails or has some errors.
The manifest files and the Dockerfile are available on the GitHub repository given below:
Also, the Groovy Script to be used is available on the GitHub repository below.
Pre-requisites
- System with Docker-CE software running. For this, I will be using a RHEL8 OS as a Virtual Machine. To install docker-ce, refer the article on this link.
- A Kubernetes Cluster. A serverless cluster can be created using Cloud services such as AWS EKS service. Or a Single-Node Cluster can be launched as a VM over the BaseOS; for this install Minikube, available to download from this link.
If using Minikube, to start the Kubernetes Single Node Cluster execute the command, minikube.exe start below in a terminal or command-line of BaseOS.
Before building the docker image for Jenkins, 4 files must be copied from Windows OS to RHEL8 VM. The four files are config, ca.crt, client.crt & client.key; config file is present in the .kube folder under root directory & the other 3 files are available in the .minikube folder under root directory. In case of Cloud services, only the config file is copied.
Copy these files to some workspace using tools like WinSCP. In the same workspace create a file name 'Dockerfile'. This file is also available on this link or on the GitHub repository, 'sarthakSharma5/repo_web' mentioned above.
FROM centos:latest RUN yum install wget -y RUN yum install sudo -y RUN yum install python38 -y RUN sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo RUN sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key RUN yum install java-11-openjdk.x86_64 -y RUN yum install jenkins -y RUN yum install git -y RUN yum install initscripts -y RUN yum install /sbin/service -y RUN sed -i "102i jenkins ALL=(ALL) NOPASSWD: ALL" /etc/sudoers # RUN echo "jenkins ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers CMD sudo service jenkins start -DFOREGROUND && /bin/bash RUN chmod 0440 /etc/sudoers RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" RUN chmod +x ./kubectl RUN mv ./kubectl /usr/bin/ RUN mkdir /root/.kube COPY ca.crt /root/ COPY client.crt /root/ COPY client.key /root/ COPY config /root/.kube CMD ["java","-jar","/usr/lib/jenkins/jenkins.war"] EXPOSE 8080
Execute the commands given below to create the docker image. Specify the complete path of Dockerfile or use '.'.
docker build -t jenkube:v1 /root/dev_task6 docker images | grep jenkube
Use docker images command to view the just created image. To launch a container and use the Jenkins WebUI use the docker run command and use "-p" option for PATting; a port is assigned to access the WebUI from a web-browser.
docker run -it -p 8085:8080 --name myjenkins jenkube:v1
The Output of the docker run command will consist of password for Admin user which will be used to access Jenkins WebUI. This password will also be available in the given file.
Use the IP address of the RHEL8 system and the port to get the WebUI and proceed with further installation; open a web-browser and use the URL: IP:8085. Sign in page opens and prompts for the same password highlighted above; to copy and use the password, open the WebUI on the VM itself if RHEL8 is installed as GUI.
Jenkins opens a Setup Wizard to install suggested plugins continue and after all are installed Getting Started page opens; create Admin user with password.
Proceed and finally the Jenkins WebUI/portal opens. Now the same WebUI can be opened from the BaseOS browser too.
Before proceeding, install two plugins on Jenkins: Job DSL and Build pipeline. For this, navigate to Manage Jenkins > Manage Plugins. Switch to Available menu and search for the 2 plugins.
On the Jenkins dashboard, use New Item to create a free-style job.
This job will get the script from the specified GitHub repository, seed.groovy as & when the Developer pushes the code. This job known as Seed Job will be used to generate the different jobs and a build-pipeline view. The script is available on the GitHub repository 'sarthakSharma5/devops_task6' mentioned above.
- Configure Job: add GitHub repository as SCM.
- Use some Build Trigger so that the script is pulled as and when changed, such as Poll SCM.
- Add a build step 'Process Job DSLs' to specify DSL scripts to use for creating Jobs & the build-pipeline view.
This script seed.groovy is a compiled script of 5 different scripts to create each of the four Jobs and the View. Each script is shown and described below.
Script for Job1:
freeStyleJob('job1-pull-repo') {
description("pull codes from GitHub")
scm {
github('sarthakSharma5/repo_web', 'master')
}
triggers {
scm("* * * * *")
}
steps {
shell(''' echo "copying code to workspace"
if sudo ls /root/ | grep task6
then
echo "dir exists"
else
sudo mkdir /root/task6
fi
sudo cp -rvf * /root/task6/
''')
}
}
This script will create a job to download the program files for webpage and copy to some workspace.
Script for Job2:
freeStyleJob('job2-deploy-code') {
description('launch app over Kubernetes Cluster')
triggers {
upstream('job1-pull-repo', 'SUCCESS')
}
steps {
shell ('''
if sudo kubectl get deploy | grep myweb
then
echo "Clearing Environment"
kubectl delete all --all
fi
if sudo ls /root/task6/ | grep .html
then
echo "Launching WebApp"
sudo kubectl apply -f htdeploy.yaml
pod=$(sudo kubectl get pods -l app=myweb -o jsonpath="{.items[0].metadata.name}")
sleep 30
sudo kubectl cp /root/task6/*.html $pod:/usr/local/apache2/htdocs
elif sudo ls /root/task6/ | grep .php
then
echo "Launching WebApp"
sudo kubectl apply -f phdeploy.yaml
pod=$(sudo kubectl get pods -l app=myweb -o jsonpath="{.items[0].metadata.name}")
sleep 30
sudo kubectl cp /root/task6/*.php $pod:/var/www/html
else
echo "appropriate file not found"
fi
''')
}
}
This Script creates the job to checks if the code is written using HTML or PHP and uses the appropriate manifest file & contact to the Kubernetes Cluster to launch a Deployment, PVC and a Service for the webpage.
2 different manifest files are used with different docker images; one for httpd and the other for PHP, the manifest files are available on the GitHub repository 'sarthakSharma5/repo_web' mentioned above.
Script for Job3:
freeStyleJob('job3-check-status') {
description('check status of the WebApp')
triggers {
upstream('job2-deploy-code', 'SUCCESS')
}
steps {
shell('''
status=$(curl -o /dev/null -s -w %{http_code} <K8S_CLUSTER>:31180/) #put IP of Minikube
if [ $status == 200 ]
then
echo "OK"
exit 0
else
echo "Error"
sudo curl --user admin:<JENKINS_ADMIN_PASSWORD> "https://<SYSTEM_IP>:8085/job/job4-mail-dev/build?token=sendmail" # Use Jenkins Admin Password
exit 1
fi
''')
}
}
This script will be creating Job3 to check if the webpage launched has errors or not using status code. If yes then a mail must be sent to the developer.
Script for Job4:
freeStyleJob('job4-mail-dev') {
description('mail developer if error in code')
authenticationToken('sendmail')
steps {
shell('''
# execute bash commands
sudo python3 /root/task6/mail.py
''')
}
}
The Script creates Job4 built/run when triggered by some other scripts, use of authenticationToken. For this a Python Code can be used or use Email Plugin. The Python Code is available on the same GitHub repository, 'sarthakSharma5/repo_web' mentioned above.
Script to create Build-Pipeline View:
buildPipelineView('DevOps Task6') {
filterBuildQueue()
filterExecutors()
title('CI/CD pipeline using Seed Job')
displayedBuilds(2)
selectedJob('job1-pull-repo')
alwaysAllowManualTrigger()
showPipelineParameters()
refreshFrequency(10)
}
This script uses the plugin installed above to create a Build Pipeline View. The view created will be titled 'CI/CD pipeline using Seed Job', will have be refreshed after every 10 secs and will be able to display 2 Builds.
Build/run the very first created job, Seed Job
For the very first build, the Job will fail. The Console Output specifies that Jenkins requires to Approval for the script.
To approve, navigate to Manage Jenkins > Security > In-process Script Approval. Select the Approve option marked below:
Re-run the Seed Job. This time the job runs successfully. The Console Output shows the generation of 4 jobs and a view according to the script.
The 4 jobs and the Build Pipeline View are generated successfully and shown on the Dashboard. The generated Jobs are shown below.
All the jobs shown above except seed_job have been created with the use of the script. Below are the images showing the configurations made to each of the four jobs automatically.
job1-pull-repo:
Gets the required manifest files and webpage code and copies to some workspace.
job2-deploy-code:
This Job is triggered when Job 1 executes completely. It checks the language used to create the webpage and launches a deployment suitable for the webpage.
The deployment will use either a httpd image if HTML is used or a PHP-installed image. Kubernetes will download the docker image from hub.docker.com behind-the-scene to be used.
To launch the deployment one of the manifest files are used from the workspace; these are also provided in GitHub repository 'sarthakSharma5/repo_web'. An example of the manifest files used, htdeploy.yaml is given below:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvweb
labels:
app: myweb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: mysvc
labels:
app: web-svc
spec:
selector:
app: myweb
type: NodePort
ports:
- nodePort: 31180
port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpdeploy
labels:
app: myweb
spec:
replicas: 1
selector:
matchLabels:
app: myweb
env: production
template:
metadata:
name: httppod
labels:
env: production
app: myweb
spec:
containers:
- name: httpcont
image: httpd
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/local/apache2/htdocs"
name: mypvc
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: pvweb
This file would help to launch three resources: a PVC, Service and a Deployment for httpd-installed pod.
job3-check-status:
This Job is triggered after a successful execution of Job 2. It will check the "Status Code" of the WebApp. The status-code returned will be equal to 200 iff the webpage shows no errors.
If status is 'not OK' then a mail is sent to the Admin or the Developer by calling Job 4 using curl command; configure Job 4 to use build trigger using scripts.
job4-send-mail:
This job is triggered by Job 3 iff there is/are some error(s) in the WebApp. It will send a mail to the specified mail address by running a python code. The code can be downloaded from the same GitHub repository, 'sarthakSharma5/repo_web'. Some changes required before using the code is also provided in the README.md.
Sample output of the mail received if Jenkins builds the job is shown below.
Start building job1 manually or wait for a trigger in a minute.
The complete pipeline runs successfully and the respective outputs of each of the jobs is given below.
job1 output:
All the files from the repository 'sarthakSharma5/repo_web' are copied to a workspace inside the container. These files include the manifest files, webpage code and the python code to mail developer. Job2 is triggered for build.
job2 output:
3 different resources are created on the Kubernetes cluster using one of the manifest file available in the workspace; a Persistent Volume Claim (PVC), Service and a Deployment. The webpage is copied to the pods and then Job3 is triggered for build.
job3 output:
Webpage launched is checked and since status code equals 200 i.e., no error hence an 'OK' message is printed and job executes successfully.
Build Pipeline View generated:
The complete pipeline resembles to the one described in the article on this link, but this time created dynamically.
The deployment launched can be view using the kubectl get all command.
The Webpage launched can be visited using the IP of Minikube; get by using the command minikube.exe ip on a terminal or command-line of BaseOS. Use a web-browser to visit the webpage on URL: MinikubeIP:31180/pagename
Use kubectl delete all --all command to delete all the resources created on the cluster.
Finally, the motive of this project is achieved. The complete CI/CD pipeline with a proper view can be created by the Developer himself or herself using the Script in Groovy language as DSL to create jobs and views in Jenkins.
Thus with complete end-to-end automation
I would like to thank my mentor Mr. Vimal Daga and LinuxWorld Informatics for guiding me with the Right Education and helping me shape my future.