Automating CI/CD Pipeline in DevOps

In this Article, We are going to Automate the Deployment and Monitoring of a Web App through the Jenkins running in Docker container. To achieve this, First of All we are going to create a Dockerfile with which we are going to build our Jenkins Image for Docker. The Dockerfile for our image is shown in the Screenshot below :-

Dockerfile is the official scripting language by docker for creating our own customized images for docker so that we won't have to build the same architecture again for the similar or same project requirements. I will explain each line in the Dockerfile and Project side by side so that you can understand everything easily.

Our Goals for the Project :-

1. Create a container image that has jenkins installed using dockerfile
2. When we launch any container from this image, it should automatically start jenkins service in the container
3. Create a job chain of job1, job2, job3, and job4 using build pipeline plugin in jenkins
4. Job1 - Pull the Github repo automatically whenever the developer push repo to Github
5. Job2 - By looking the type of files in the repo (php, html etc.) jenkins should automatically start the respective server and deploy that code on that server
6. Job3 - It should test that the Web App or Code is deployed on the server or not
7. Job4 - If the code is not deployed on the server, it should send an email to the developer
8. Job5 - It should keep monitoring our container which has the server running of our website or Web app and if by any chance, the container fails or stops, it should automatically start the container again

Let's Start with what we want to Achieve :-

1. We have to create a jenkins image for docker using dockerfile

We need an image to run our program and to run our program we need an Operating System which will have all the dependencies needed by our program to run. That's why We have written in the first line of dockerfile - [FROM centos] which means that we will be using latest version image of centos i.e. centos 8 or centos:latest as the base OS for installing jenkins and for installing jenkins we need to download jenkins repo and import the jenkins key in our yum repositories of centsos and to download the repo we will need wget in our OS, that is the reason we have used - [RUN yum install wget -y] and wget will download the repo in yum repos [RUN wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo] and then we will import the key [RUN rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key]

Now we have everything to download and install jenkins in our centos or jenkins image, we can directly run [yum install jenkins] but that always got stuck like infinite whenever we tried to build the image using our Dockerfile, as shown in screenshot below (P.S - I had tried more than 100 times literally to solve this problem using --no-cahe in build, tried gpgcheck=0 in every repo, replaced centos with centos:latest, restarted docker, installed java before installing jenkins etc. everything suggested by every source but it remained same and at the end it always showed an error - couldn't download tried every mirror - I guess this was because maybe jenkins servers are slow)

Now to solve this error, I didn't sleep one whole night and finally came up with a solution for this, it may look very simple but this is the only thing that worked and guaranteed that jenkins will be installed in every build. To solve this, I downloaded the rpm locally on my docker host and then copied inside the centos image to install locally, you can also use wget to download the rpm during build but sometimes it may also get stuck and if your build has some error after that step then you will have to do that again so better use copy because you have to download only once and you can use it as many times as you want, after the installation, you can remove the rpm file from the image during build, that's why I have used these things in dockerfile - [COPY jenkins-2.222.3-1.1.noarch.rpm /, RUN yum install jenkins-2.222.3-1.1.noarch.rpm -y, RUN rm -f jenkins-2.222.3-1.1.noarch.rpm]

Now we have successfully installed jenkins in our image, we will move to our next step now but before that I will explain you the lines which are left in our dockerfile.

We have used [RUN yum install git -y] to perform git operations inside jenkins shell and [RUN yum install python36 -y] for additions help if we may need to automate something using python in future from jenkins shell and [RUN echo "jenkins ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers] to set jenkins as sudo user to skip password for performing any operation in the command line.

2. Whenver we launch our Image, it should automatically launch jenkins for us

Before moving to that part, we know that we will have to execute some commands from the docker host OS to start and monitor the containers, for that we are going to use ssh but password less (using keys so that we can easily run our commands) and for that we need to configure in our container, to configure, we need to install ssh first in our image, so for that we have used [RUN yum install openssh-server -y & RUN yum install openssh-clients -y] but the problem here is we can't pass the keys simply in our image as the image is used by multiple people and we don't want to compromise our security. Also, everyone will have their own username and ip to connect so image should be general for all and people should be able to specify their own username and ip address to connect using ssh.

So to achieve this thing I have used [ENV USER=root] so that we can specify username for ssh during runtime and if we don't specify this, by default it will use root and [ENV DEST_IP $DEST_IP] so that we can enter the ip during runtime whenever we launch the image it will automatically start configuring ssh for us. But the problem here is we have to set jenkins to start at the launch of container and we can't specify multiple commands for the start, we can only specify one command, even if we specify multiple commands, it will only execute the one given at the last. So, here we will use a trick, we will put all the commands in a file and execute it using bash (technically we executed only 1 command at the start) that's why We have used - [COPY script /] to first of all copy the script to the container and then [CMD bash script] execute it at the start.

The script looks like this as shown in the screenshot below :-

Firstly, it will generate the key and then it will generate the key, then it will copy the key to the system of whichever ip you want to connect and then it will show some help for the people who maybe new to ssh using our image and at the end it will start our jenkins and whenever you restart the container, it will automatically skip the keys and ssh configuration part because it will detect that the keys are already there. Now, Jenkins need openjdk-java to run, so that's why we have also installed java-11 in our image using - [RUN yum install java-11-openjdk -y]

Now, we will build our image with the name jenkins:lts in the current directory where we have kept our Dockerfile using the command - docker build -t jenkins:lts . as shown in screenshot below :-

Now, after our image is built, let's run our very first container from the image, for this, just run the command - docker run -it --name jen1 -v /home/dheeth/jenkins_vol/:/root/.jenkins/ -p 8082:8080 -e USER=jenkins -e DEST_IP=192.168.100.3 jenkins:lts as shown in the screenshot below :-

We have used a volume so that if we lose the container, we won't lose our data as we can connect the same volume to the new container and as soon as we launch it, it will start configuring our ssh for [email protected] because we have already given sudo permissions without password to jenkins in our docker host OS. After that, jenkins will be started as shown in the screenshot below :-

Unlock your jenkins using the password you will get in the command line, Install Github and Build Pipeline plugins and Let's move to our Jobs :-

3. Create 4 Jobs using build pipeline View

Create 4 jobs namely job1, job2, job3 & job4 and we add it to a new build pipeline view as shown in screenshot below, We will add the functioning further :-

4. Job1 - Pull the Github repo automatically whenever the developer push repo to Github

For this, just setup a github webhook which will automatically trigger our job1 and give the github repository url using Github plugin installed to the job1 and setup build trigger by Github hook trigger and after that transfer all the files downloaded from github to our directory from where our webapp will be deployed on the server but copy only if we receive php or html files otherwise there is nothing to deploy, from now onwards we will use ssh for running our commands in docker host os as shown in screenshots below :-

5. Job2 - By looking the type of files in the repo (php, html etc.) jenkins should automatically start the respective server and deploy that code on that server

For this, set this job to trigger after job1 and just check if your container is running already, remove it then if you have php files in the directory, run the container from php interpreter image and if only html files are there in directory, then run httpd server only as shown in screenshot below :-

6. Job3 - It should test that the Web App or Code is deployed on the server or not

For this, set this job to trigger after job2 is built and then just check the status code of your deployed web app url, if you get 200 that is ok, else the job should fail as shown in screenshot below :-

7. Job4 - If the code is not deployed on the server, it should send an email to the developer

For this, just setup your smtp server with whatever you feel comfortable, here in our case we will use sendinblue free smtp server for this and then check the last build status of the job3 using jenkins API and set it to trigger after job3 even if the build is unstable. I have checked for result (SUCCESS word) in the API and counted it, basically set to 1 or 0 and if received 1 that means last build of job3 was successful and if not send an email for unstable build by post build triggers as shown in screenshots below :-

8. It should keep monitoring our container which has the server running of our website or Web app and if by any chance, the container fails or stops, it should automatically start the container again

For this, you can check if your container was launched or not and if it was launched then keep checking if your container where your web app is deployed is running or not and that is your wish you can set it to every minute or every hour, here we have used every minute in our case. You can check if your container is showing in both the list of running and all containers, if showing only in all and not in running, then start the container again as shown in screenshot below :-

Our Pipeline is ready to run now :-

and Finally we have completed our 2nd Task given by Vimal Daga sir in the DevOps Assembly Lines training under LinuxWorld Informatics pvt. ltd. I would like to thank Sir for always motivating us that we are the best and keep researching each and everytime to think like the Creators of the Technology. I would also like to thank everyone for sparing their time to read my work described in such a long article.