Jenkins

Jenkins is a free and open source automation server. It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat. 

It supports version control tools, including AccuRev, CVS, Subversion, Git, Mercurial, Perforce, ClearCase and RTC, and can execute Apache Ant, Apache Maven and sbt based projects as well as arbitrary shell scripts and Windows batch commands.

Some of the possible steps that can be performed using Jenkins are:

• Software build using build systems such as Gradle, Maven, and more.
• Automation testing using test frameworks such as Nose2, PyTest, Robot, Selenium, and more.
• Execute test scripts (using Windows terminal, Linux shell, etc.)
• Achieve test results and perform post actions such as printing test reports, and more.
• Execute test scenarios against different input combinations for obtaining improved test coverage.
• Continuous Integration (CI) where the artifacts are automatically created and tested. This aids in identification of issues in the product at an early stage of development.

Features of Jenkins

• Easy Installation & Configuration
• Open source; free for use
• Thriving Plugin ecosystem
• Easy distribution

Architecture Of Jenkins

These are the series of steps that outlines the interaction between different elements in Jenkins:

• Developers do the necessary modifications in the source code and commit the changes to the repository. A new version of that file will be created in the version control system that is used for maintaining the repository of source code.
• The repository is continuously checked by Jenkins CI server for any changes (either in the form of code or libraries) and changes are pulled by the server.
• In the next step, we ensure that the build with the ‘pulled changes’ is going through or not. The Build server performs a build with the code and an executable is generated if the build process is successful. In case of a build failure, an automated email with a link to build logs and other build artifacts is sent to the developer.
• In case of a successful build, the built application (or executable) is deployed to the test server. This step helps in realizing continuous testing where the newly built executable goes through a series of automated tests. Developers are alerted in case the changes have caused any breakage in functionality.
• If there are no build, integration, and testing issues with the checked-in code, the changes and tested application are automatically deployed to the Prod/Production server.

A single Jenkins server might not be sufficient to realize the following requirements:

• Testing needs to be performed on different environments (i.e. code written using different languages e.g. Java, Python, C, etc. are committed to the version control system), where a single server might not suffice the requirement.
• A single Jenkins server might not be sufficient to handle the load that comes with large-scale software projects.

In such scenarios, the distributed (or Master-Agent) architecture of Jenkins is used for continuous integration and testing. Diving deeper into how does Jenkins works, we take a look at the architecture of Jenkins.

Master- Agent Architecture In Jenkins

The master-agent (or distributed) architecture in Jenkins is used for managing distributed builds. The Master and Agent(s) communicate through the TCP/IP protocol.

These are the roles and responsibilities of the Jenkins Master and Agent(s):

Jenkins Master

The main server in Jenkins is the Master. Here are the jobs handled by Jenkins Master:

• Schedule build jobs
• Choosing the appropriate agent in the master-agent ecosystem for dispatching the builds.
• Monitor agents and take them online/offline as and when required.
• Presenting the build results (and reports) to the developer.

The Jenkins master can also execute the jobs directly but it is always recommended to select the appropriate agent(s) for build and execution-related tasks.

Jenkins Agent(s)

A agent is a remote machine that is connected to the Master. Depending on the project and build requirements, you could opt for ‘N’ number of agents. agents can run on different operating systems and depending on the ‘type of build request’, the appropriate Agent is chosen by the Master for build execution and testing.

Here are the jobs handled by the Jenkins Agent(s):

• Listen to commands from the Jenkins Master.
• Execute build jobs that are dispatched by the Master.
• Developers have the flexibility to run the build and execute tests on a particular agent or a particular type of Agent. The default option is Jenkins Master selecting the best-suited Agent for the job.

Jenkins Pipeline

Pipeline in Jenkins is a group of jobs (or events) that are interlinked in a particular sequence. Jenkins Pipeline is a set or suite of plugins that provides support for implementation and integration of Continuous Delivery pipelines into Jenkins.

The Pipeline also provides a set of tools that are useful for modeling simple as well as complex delivery pipelines ‘as code’ through ‘Pipeline Domain-Specific Language (DSL)’ syntax.

Every job in the Jenkins pipeline has some dependency on one or more events. Continuous delivery pipeline in Jenkins consists of four states – Build, Deploy, Test, and Release. Each of these states consist of events that execute in a sequence.

Jenkinsfile

Now that you understand what is Jenkins pipeline, we can dive deeper into the concept. The entire definition of a Jenkins Pipeline is written into a text file called Jenkinsfile. It contains the steps required for running a Jenkins Pipeline. ‘Pipeline as code’ can be implemented using Jenkinsfile and Domain Specific Language (DSL) is used for defining the same.

Jenkinsfile can also be committed to the source control repository of the project. With Jenkinsfile, the CD Pipeline is also treated as a part of the application that is versioned, committed, and reviewed like any other piece of code.

Some of the major benefits of Jenkinsfile are:

• Single Jenkinsfile can be used for creating a Pipeline build process for all the branches and executing pull requests.
• The implementation in a Pipeline can be reviewed like normal source code.
• Audit trail of the Pipeline.
• Singular source for the Pipeline can be viewed as well as edited by multiple members associated with the project.

Though Pipeline can be defined either in web UI or with a Jenkinsfile, it is recommended to define the what is Jenkins Pipeline in a Jenkinsfile and maintain the same in a source control repository.

Advantages Of Using Jenkins Pipeline

Jenkins Pipeline is instrumental in adding a rich set of automation tools onto Jenkins. Hence, Jenkins can be used for simple continuous integration as well as for comprehensive CD pipelines.

These are some of the primary reasons for using the Jenkins Pipeline:

• As the Jenkins Pipeline is implemented as code, it can be checked in the source control repository. Teams can view, edit, as well as iterate upon the delivery pipeline.
• Jenkins Pipelines are robust. A Pipeline is automatically resumed in case the server witnesses an unplanned restart.
• Pipeline process can pause and wait for input for any input from the user.
• Pipelines are versatile as they can be used for realizing complex CD requirements, including performing work in parallel.
• Jenkins Pipelines are extensible by plugin developers and users with Pipeline Shared Libraries.

How Jenkins works?

Jenkins runs as a server on a variety of platforms including Windows, MacOS, Unix variants and especially, Linux. It requires a Java 8 VM and above and can be run on the Oracle JRE or OpenJDK Usually, Jenkins runs as a Java servlet within a Jetty application server. It can be run on other Java application servers such as Apache Tomcat. More recently, Jenkins has been adapted to run in a Docker container. There are read-only Jenkins images available in the Docker Hub online repository.

To operate Jenkins, pipelines are created. A pipeline is a series of steps the Jenkins server will take to perform the required tasks of the CI/CD process. These are stored in a plain text Jenkinsfile. The Jenkinsfile uses a curly bracket syntax that looks similar to JSON. Steps in the pipeline are declared as commands with parameters and encapsulated in curly brackets. The Jenkins server then reads the Jenkinsfile and executes its commands, pushing the code down the pipeline from committed source code to production runtime. A Jenkinsfile can be created through a GUI or by writing code directly.

Case Study of Topdanmark

Topdenmark:

Topdanmark, a Danish insurance company offering a customer service platform that allows clients to easily and quickly manage their insurance, investments and pensions. 

Delivering tools and infrastructure to drive innovation

Since the company relies heavily on IT-infrastructure, it is imperative that every bit just works.

With roughly 400 IT-related employees in the company and just about 80 on the operations team, the rest of the developers support approximately 2,000 other employees. Within the Ops group, There are six-man DevTools team who maintain the server platform that runs most of the company’s client/customer-related business applications. They keep approximately 1,000 servers alive. The servers are put into different categories, depending on which environment they live in.

Building automation with a highly-configurable Jenkins platform

Most of Topdanmark’s servers and all their desktops are virtualized, so they are easy to replace if something breaks down. In terms of Jenkins installations, they are all virtual. 

Topdanmark has two Jenkins setups, legacy and CI/CD. In legacy setup, They have test, integration, release, and production environments. In CI/CD setup, they have a non-production and a production environment.

The legacy setup is part of a pre-scheduled handheld deployment cycle, customized to whichever team uses it. Due to all the customizations, it could take several days from the developer request to having a working Jenkins instance. This is ofcourse take several time but using CI/CD will make it easy and fast.

The new setup is automated and easier to maintain. For this new iteration, the team took a different, more modern approach to deliver the service to its users. They created a self-service portal that allows anyone within the company to ‘order’ a Jenkins instance, a huge timesaver.

Driving developer success with automation and responsiveness

Due to the very nature of development and a constant flow of business needs, it’s imperative for organizations to support developers by supplying tools and solutions that allow them to create the software needed to solve the task at hand. 

The secret behind this is that all the new Jenkins instances are based on spinning up a custom Jenkins Docker image, bootstrapping some multi-branch configuration like predefined plugins and creating the multi branch repositories from one or more GitHub organizations that are part of the request-form.

To simplify the user experience even further, Topdanmark created quite a few Groovy scripts that can be used in the Jenkinsfiles. They basically hide all the complicated stuff but maintain control regarding internal audits, and other corporate rules.

Evolving tools, evolving applications

With the help of Jenkins, Topdanmark keeps tooling up to date. They are in continuous dialogue with the developers regarding what they want and need to do their job. By continuously evolving the tools, they better support them as they continue to evolve their applications. 

Thank you!