DevOps Automation using Groovy in Jenkins Job DSL

Whenever we talk about DevOps, it comprises of 2 words Dev + Ops meaning Developers sharing some load of the Operations staff in an organization. In this article, I have tried to show a small demo of it with a real world example. We are going to create, deploy and test a web server running on top of Kubernetes with a single DSL (Domain Specific Language) of jenkins i.e. groovy. Now let's directly jump to the main part of it's execution.

How will it work :-

Our operations staff will create one job which will run the DSL script shared by the developer using Github code etc. and that script will create all the other jobs required for the deployment, testing or anything with all the specifications. The job first created by operations staff which generates other is will be called seed-job. Now let's jump to the practical part.

Problem Statement :-

Our problem statement is same as of previous article Automating Continuous Deployment in DevOps using Jenkins (Refer to this article if you don't userstand anything, complete explanation of commands is used inside shell is written in this article).

1. Create container image that has Jenkins installed using dockerfile. When we launch this image, it should automatically start Jenkins service in the container.
2. Create a job chain of job1, job2, job3 and job4 using build pipeline plugin in Jenkins

Job1 : Pull the Github repo automatically when some developers push repo to Github.

Job2 : 

1. By looking at the code or program file, Jenkins should automatically start the respective language interpreter installed image container to deploy code on top of Kubernetes ( eg. If code is of PHP, then Jenkins should start the container that has PHP already installed )

2. Expose your pod so that testing team could perform the testing on the pod

3. Make the data to remain persistent ( If server collects some data like logs, other user information )

Job3 : Test your Website if it is working or not.

 Job4 : if Website is not working , then send email to developer with error messages and redeploy the application after code is edited by the developer.

Now we have already done the same task in the article given above and creating an image with jenkins installed and passwordless ssh setup is also explained by me in Automating CI/CD Pipeline in DevOps. The only difference is, previously we created all the jobs manually but this time we are going to create all the jobs automatically using jenkins job dsl so we are going to focus more on that.

Initial Steps :-

For the start, download and install Job DSL plugin in jenkins and create a job called seed-job. In this job, we can either mention the groovy filename with .groovy extension or we can directly write our DSL script inside it which will automatically generate the other jobs. We are going to follow the direct script writing approach.

Job to pull github repository :-

The first job which will be created by seed-job is given the name git-pull because it will clone our github repository which has all the files needed for deployment into workspace. Download and install Github plugin to continue with this. The first requirement is to provide the link of github repo (scm). Also we want to setup a trigger to run the job with every git push to github using github-webhooks. Then we need to run some commands in our shell (steps-shell) and our first job completes here. The code for the first job is :-

job("git-pull"){
  description("This job will pull our files from github")
  scm{
    github('dheeth/kube-web', 'master')
  }
  triggers{
    githubPush()
  }
  steps{
    shell('''
ssh [email protected] '
path="/home/ec2-user/jenkins_vol/workspace/git-pull/"
files=$(ls $path | grep -e php -e html | wc -l)
if [[ $files > 0 ]]
then
cp $path* /home/ec2-user/master
else
echo "No php or html files received"
exit 1
fi
'
''')
  }

}

Job to deploy our code on top of Kubernetes :-

For this, I have connected a cluster of Kubernetes from https://okteto.com because my laptop doesn't have enough resources to run K8s in my local system. So, this job (launch-web) needs to be triggered after the git-pull and then we'll execute our commands in the shell for the deployment of our code.

job("launch-web"){
  description("This job will deploy our website on top of K8s")
  triggers {
    upstream('git-pull', 'SUCCESS')
  }
  steps{
    shell('''
ssh [email protected] '
html_files=$(ls /home/ec2-user/master | grep html | wc -l)
php_files=$(ls /home/ec2-user/master | grep php | wc -l)


if kubectl get deploy | grep myweb
then
kubectl delete deploy myweb
else
echo "No myweb deployment found, you can proceed to next step"
fi


if [[ $html_files > 0 && $php_files > 0 ]]
then
sudo docker build --build-arg IMAGE=php:7.2-apache --build-arg FILES=* --build-arg DIR=/var/www/html -t dheeth/kube-web /home/ec2-user/master
elif [[ $html_files > 0 ]]
then
sudo docker build --build-arg IMAGE=httpd --build-arg FILES=*.html --build-arg DIR=/usr/local/apache2/htdocs -t dheeth/kube-web /home/ec2-user/master
elif [[ $php_files > 0 ]]
then
sudo docker build --build-arg IMAGE=php:7.2-apache --build-arg FILES=*.php --build-arg DIR=/var/www/html -t dheeth/kube-web /home/ec2-user/master
fi


sudo docker push dheeth/kube-web


kubectl create -f /home/ec2-user/master/pvc.yml
kubectl create -f /home/ec2-user/master/deploy.yml
sleep 10
kubectl expose deploy myweb --type=LoadBalancer --port=80
sleep 10
kubectl get svc
'
''')
  }

}

Job to test our website running on the webserver :-

Now here I have a fixed url for my website until the deployment is there by okteto cloud, I get the status code from there to test if the website is up and running or not. Now this job (test-web) will be triggered after the previous job (launch-web) and then here again we need to run some commands in the shell. So the code for this job is :-

job("test-web"){
  description("This job will test our website on top of K8s")
  triggers {
    upstream('launch-web', 'SUCCESS')
  }
  steps{
    shell('''
ssh [email protected] '
status=$(sudo curl -o /dev/null -sw "%{http_code}" https://myweb-dheeth.cloud.okteto.net)


if [[ $status == 200 ]]
then
echo "Website is up and running"
else
echo "Something Went Wrong"
exit 1
fi
'
''')
  }

}

Job to notify Developer :-

Jenkins will get the status of lastBuild from it's api of the previous job (test-web) that it's SUCCESS or FAILURE, and If website is not running it should send an email to the developer about the build status with some logs. So, this job (notify) will be triggered by the previous job even if that job fails and it will get the status by running some commands in the shell and then after post build, it will send an email to the developer (publlishers - mailer).

job("notify"){
  description("This job will deploy our website on top of K8s")
  triggers {
    upstream('test-web', 'FAILURE')
    scm("* * * * *")
  }
  steps{
    shell('''
ssh [email protected] '
status=$(curl -s https://admin:12%[email protected]/job/test-web/lastBuild/api/xml | grep SUCCESS | wc -l)


if [[ $status == 1 ]]
then
echo "Website is up and running"
else
echo "Something went wrong, Sending email to developer"
exit 1
fi
'
''')
  }
  publishers{
    mailer('[email protected]', false, false)
  }

}

Now when we run our seed-job, it generates 4 more jobs using DSL code by the developer :-

The jobs are automatically created and configured exactly how we wanted them to be using the DSL code of jenkins as shown in screenshots below :-

The console logs after the jobs were triggered by the github-webhook, were also exactly the same as the previous one in which we configured the jobs ourselves :-

and here is our Website Up and Running which is deployed on okteto cloud :-

I hope this article will help you to achieve what you have always wanted and if you still didn't understand anything you can message me on my linkedin profile or comment below. The main focus in this article was mainly on the DSL because explanations are already done in the previous articles which are mentioned above. If you liked this post, do like and share with your friends and colleagues so that they can also learn a new approach.