CI CD pipeline with Docker in automation
QAP Software Solutions
IT Staffing and Recruiting, Software Development & QA services. www.qapsoftwaresoftware.com
?If you’re still?building and delivering your software applications the traditional way then you are missing out on a major innovation in the Software Development Process or Software Development Life Cycle. To show you what I’m talking about, in this article I will share “How to create a CI CD Pipeline with Jenkins, Containers and Amazon ECS” that deploys your application and overcomes the limitations of the traditional software delivery model. This innovation greatly affects deadlines, time to market, quality of the product, etc. I will take you through the whole step-by-step process of setting up a CI CD Docker pipeline for a sample Nodejs application.?
Table of contents
A CI CD Pipeline or Continuous Integration Continuous Delivery Pipeline is a set of instructions to automate the process of Software tests, builds, and deployments. Here are a few benefits of implementing CI CD in your organization.
1.Smaller Code Change
The ability of CI CD Pipelines to allow the integration of a small piece of code at a time helps developers to recognize any potential problem before too much work is completed.
2.Faster Delivery
Multiple daily releases or continual releases can be made a reality using CI CD Pipelines.
3.Observability
Having automation in place that generates extensive logs in each stage of the development process helps to understand if something goes wrong.
4.Easier Rollbacks
There are chances that the code that has been deployed may have issues. In such cases, it is very crucial to get back to the previous working release as soon as possible. One of the biggest advantages of using the CI CD Pipelines is that you can quickly and easily roll back to the previous working release.
5.Reduce Costs
Having automation in place for repetitive tasks frees up the Developer and Operation guys’ time that could be spent on Product Development.
These are just a few benefits of having CI CD Pipelines for builds and deployments. On this video you can continue learning the CI CD benefits and why use a CI CD in place.
?Now, before we proceed with the steps to set up a CI CD Pipeline with Jenkins, Containers, and Amazon ECS, let’s see in short what tools and technologies we will be using.?
?
CI CD Docker Tool Stack
?Github
It is a web-based application or a cloud-based service where people or developers collaborate, store and manage their application code using Git. We will create and store our sample Nodejs application code here.
?
Architecture
This is how our architecture will look like after setting up the CI CD Pipeline with Docker.
After the CI CD Docker Pipeline is successfully set up, we will push commits to our Github repository and in turn, Github Webhook will trigger the CI CD Pipeline on Jenkins Server. Jenkins Server will then pull the latest code, carry out unit tests, build a docker image and push it to AWS ECR. After the image is pushed to AWS ECR, the same image will be deployed in AWS ECS by Jenkins.
CI CD workflow and phases
Workflow
CI and CD Workflow allows us to focus on Development while it carries out the tests, build, and deployments in an automated way.?
Phases
The primary goal of the automated CI CD pipeline is to build the latest code and deploy it. There can be various stages as per the need. The most common ones are mentioned below.
Deployment strategy
A deployment strategy is a way in which containers of the micro-services are taken down and added. There are various options available; however, we will only discuss the ones that are available and supported by ECS
Rolling updates
In rolling updates, the scheduler in the ECS Service replaces the currently running tasks with new ones. The tasks in the ECS cluster are nothing but running containers created out of the task definition. Deployment configuration controls the number of tasks that Amazon ECS adds or removes from the service. The lower and the upper limit on the number of tasks that should be running is controlled by minimumHealthyPercent and maximumPercent respectively.?
If you want to learn more about this, visit the official documentation?here.
Blue/Green Deployment
Blue/Green deployment strategy enables the developer?to verify a new deployment before sending traffic to it by installing an updated version of the application as a new replacement task set.
There are primarily 3 ways in which traffic can shift during blue/green deployment.
To learn more about this, visit the official documentation?here.
Out of these 2 strategies, we will be using the rolling-updates deployment strategy in our demo application.
?
Dockerize Node.js app
Now, let’s get started and make our hands dirty.
The Dockerfile for the sample Nodejs application is as follows. There is no need to copy-paste this file, it is already available in the sample git repository that you cloned previously.
Let’s just try to understand the instructions of our Dockerfile.
This is the Docker file that we will use to create a docker image.
Setup Github Repositories
Create a new repository
Note: Create a new text file on your system and note down all the details that will be required later.
Create a Github Token
This will be required for authentication purposes. It will be used instead of a password for Git over HTTPs, or can be used to authenticate to the API over Basic Authentication.
2. Click on the?“Personal access tokens” options and “Generate new token” to create a new token.
3. Tick the “repo” checkbox, the token will then have “full control of private repositories”
4. You should see your token created now.
Clone the sample Repository
Note:?You are in the home directory, i.e. /home/ubuntu.
Note:?For the rest of the article, do not change your directory. Stay in the same directory, here it is /home/ubuntu/demo-nodejs-app/, and execute all the commands from there.?
Push your first commit to the repository
Setup the AWS Infrastructure
Create an IAM User with a Programmatic Access
Create an ECR Repository
Create an ECS Cluster
2. Click on the “Configure” button available in the “custom” option under “Container definition”.
3. Specify a name to the container as “nodejs-container”, the ECR Repository URL in the “Image” text box, “3000” port in the Port mappings section, and then click on the “Update” button. You can specify any name of your choice for the container.
4. You can now see the details you specified under “Container definition”. Click on the “Next” button to proceed
5. Select “Application Load Balancer” under “Define your service” and then click on the “Next” button.
6. Keep the cluster name as “default” and proceed by clicking on the “Next” button. You can change the cluster name if you want.
7. Review the configuration and it should look as follows. If the configurations match, then click on the “Create” button. This will initiate the ECS Cluster creation.
8. After a few minutes, you should have your ECS cluster created and the Launch Status should be something as follows.
Create an EC2 Instance for setting up the Jenkins Server
Setup Jenkins on the EC2 Instance
After the instance is available, let’s install Jenkins Server on it along with all the dependencies.
Pre-requisites of the EC2 Instance
Install Java, JSON Processor jq, Nodejs/NPM and aws-cli on the EC2 Instance
Note: Make sure all your versions match the versions seen in the above image.?
Install Jenkins on the EC2 Instance
Install Docker on the EC2 Instance
Configure the Jenkins Server
3.Select the “Install suggested plugins” option
4. Specify user-name, password for the new admin user to be created. You can use this user as an admin user.
5. This URL field will be auto-filled, click on the “Save and Finish” button to proceed.
6. Your Jenkins Server is ready now.
7. Here is how its Dashboard looks like.
Install Plugins
4. You will see the screen as follows after the plugins have been installed successfully.
Create Credentials in Jenkins
2. Click on “(global)” “Add credentials”.
3. Select Kind as “AWS Credentials” and provide ID as “demo-admin-user”. This can be provided as per your choice,?keep a note of this ID in the text file. Specify the Access Key and Secret Key of the IAM user we created in the previous steps. Click on “OK” to store the IAM credentials.?
4. Follow the same step and this time select Kind as “Username with password” to store the Github Username and Token we created earlier. Click on “Ok” to store the Github credentials.
5. You should now have IAM and Github credentials in your Jenkins.
Create a Jenkins Job
2. Select the “Pipeline” and name it as “demo-job” or provide a name of your choice.
3. Tick the “Github project” checkbox under the “General” tab, provide the Github Repository URL of the one we created earlier. Also, tick the checkbox “Github hook trigger for GitScm polling” under the “Build Trigger” tab.
4. Under the “Pipeline” tab, select “Pipeline script from the SCM” definition, specify our repository URL and select the credential we created for Github. Check the branch name if it matches the one you will be using for your commits. Review the configurations and click on “Save” to save your changes to the pipeline.
5. Now you can see the pipeline we just created.
Integrate Github and Jenkins
The next step is to integrate Github with Jenkins so that whenever there is an event on the Github Repository, it can trigger the Jenkins Job.
2. Provide the Jenkins URL with context as “/github-webhook/”. The URL will look as follows.
Webhook URL:?https://<Jenkins-IP>:8080/github-webhook/
You can select the events of your choice; however, for the sake of simplicity I have chosen “Send me everything”. Make sure the “Active” checkbox is checked. Click on “Add webhook” to create a webhook that will trigger the Jenkins job whenever there is any kind of event in the Github Repository.
3. You should see your webhook. Click on it to see if it has been configured correctly or not.
4. Click on the “Recent Deliveries” tab and you should see a green tick mark. The green tick mark shows that the webhook was able to connect to the Jenkins Server.
Deploy the Nodejs Application to the ECS Cluster
Before we trigger the Pipeline from Github Webhook, lets try to manually execute it.
Build the Job Manually
2. If you see its logs, you will see that it failed. The reason is, we have not yet assigned values to variable we have in our Jenkinsfile
Push your second commit
Reminder Note:?For the rest of the article, do not change your directory. Stay in the same directory i.e. /home/ubuntu/demo-nodejs-app and execute all the commands from here.?
Assign values to the variable in the Jenkinsfile
2. Here is the list of variables for your convenience.
We have the following variables in the Jenkinsfile.
3. Check the status to confirm that the file has been changed.
git status
cat Jenkinsfile
4. Add a file to the git staging area, commit it and then push to the remote Github Repository.
git status
git add Jenkinsfile?
git commit -m “Assigned environment specific values in Jenkinsfile”
git push
Error on Jenkins Server
After pushing the commit, the Jenkins Pipeline will get triggered. However, you will see an error “Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock” in your Jenkins Job. The reason for this is, a “jenkins” user that is used by the Jenkins Job is not allowed to create docker objects. To give permission to a “jenkins” user, we added it to the “docker” group in the previous step; however, we did not restart the Jenkins service after that.?
I kept this deliberately so that I could show you the need to add the “jenkins” user to the “docker” group in your EC2 Instance.
Now you know what needs to be done to overcome the above error.
Push your third commit
3. This time you can observe that the job must have got triggered automatically. Go to the Jenkins job and verify the same.
4. This is how the Stage View looks like. It shows us the stages that we have specified in our Jenkinsfile.
Check Status of the Task in the ECS Cluster
2. Click on the “JSON” tab and verify the image, the image tag should match with the Jenkins Build number. In this case, it is “6” and it matches with my Jenkins Job Build number.
3. Hit the ELB URL to check if the Nodejs application is available or not.You should get the message as follows in the browser after hitting the ELB URL.
Push your fourth commit
Note: Your revision numbers may differ.
2.Wait for around 2-3 minutes, and you should only have one task running with the latest revision.
3. Again hit the ELB URL and you should see your changes. In this case, we had changed the display message.
Congratulations! You have a working Jenkins?CI CD Pipeline?to deploy your Nodejs containerized application on AWS ECS whenever there is a change in your source code.
Cleanup the resources we created
If you were just trying out to set up a CI CD pipeline to get familiar with it or for POC purposes in your organization and no longer need it, it is always better to delete the resources you created while carrying out the POC. As part of this CI CD pipeline, we created a few resources. We created the below list to help you delete them.
Summary
And finally, here is the summary of what you have to do to set up a CI CD Docker pipeline to deploy a sample Nodejs application on AWS ECS using Jenkins.?
Conclusion
A CI CD Pipeline serves as a way of automating your software applications’ builds, tests, and deployments. It is the backbone of any organization with a DevOps culture. It has numerous benefits for software development and it boosts your business greatly.??
In this blog, we demonstrated the steps to create a Jenkins CI CD Docker Pipeline to deploy a sample Nodejs containerized application on AWS ECS. We saw how Github Webhooks can be used to trigger the Jenkins pipeline on every push to the repository which in turn deploys the latest docker image to AWS ECS.
CI CD Pipelines with Docker are best for your organization to improve code quality and deliver software releases quickly without any human errors. We hope this blog helped you to learn more about the integral parts of the CI CD Docker Pipeline.
FAQ
What is a CI CD Pipeline?
In simple words, a CI CD Pipeline is a set of instructions executed in order to deliver a newer version of your software application.
Why should I use a CI CD Pipeline?
To avoid human errors, enhance code quality, improve release cycles, add technical and business values, improve customer satisfaction, and much more.?
Which are the basic stages of any CI CD Pipeline?
Any basic CI CD Pipeline typically includes – Pull, Test, Build, Push and Deploy stages.
Does setting up a CI CD Pipeline need any Programming knowledge?
Not really. If you are working with a tool like Jenkins that has GUI, you can simply create a CI CD Pipeline by making some configurations on the UI. You may need some basic understanding of scripting to automate your tasks or execute commands.
What is a CI CD Docker Pipeline?
A CI CD Pipeline that tests, builds, and deploys Docker Containers of your applications is referred to as CI CD Docker Pipeline.?
Why should I use CI CD Pipeline with Docker?
One can use CI CD Pipeline with Docker to specifically automate the Software Development process and improve release cycles of microservices-based applications.
?
if u need more help please contact us at +91- 93 92 91 89 89