Jenkins Server setup with dynamic worker nodes

This article is a concise guide for creating a Jenkins Cluster which can dynamically provision worker nodes for advantages like savings in cost and parallel execution of jobs using multiple nodes. A bit of experience in working with EC2 inside AWS is required to follow the guide.

Introduction

The steps below use AWS for creating virtual machines and the EC2 plugin inside Jenkins for provisioning worker nodes. The operating system used here for the nodes is Ubuntu18.04. However, the concept can be extended to any other OS such as CentOS, AmazonLinux, etc.

Step 1: Jenkins master node setup

• Provision an EC2 with Ubuntu18.04
• SSH into the EC2.

chmod 400 jenkins_master_node_pem.pem
ssh -i jenkins_master_node_pem.pem [email protected]
        

• As a prerequisite for Jenkins - Java 8 needs to be installed.

ubuntu@ip-172-31-19-110:~$ sudo apt update -y
ubuntu@ip-172-31-19-110:~$ sudo apt install openjdk-8-jdk openjdk-8-jre -y
ubuntu@ip-172-31-19-110:~$ java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~18.04-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
        

• Install Jenkins Server

ubuntu@ip-172-31-19-110:~$ wget -q -O - https://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -

OK

ubuntu@ip-172-31-19-110:~$ sudo sh -c 'echo deb https://pkg.jenkins-ci.org/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

        

Run "sudo apt update -y". It might throw an error like below...

ubuntu@ip-172-31-19-110:~$ sudo apt-get update?

Hit:1 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease? ? ? ? ? ? ? ? ? ? ? ? ? ??
Hit:3 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease? ? ? ? ? ? ? ? ? ? ? ? ??
Ign:4 https://pkg.jenkins-ci.org/debian binary/ InRelease? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Get:5 https://pkg.jenkins-ci.org/debian binary/ Release [2044 B]? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Get:6 https://pkg.jenkins-ci.org/debian binary/ Release.gpg [833 B]? ? ? ? ? ? ??
Hit:7 https://security.ubuntu.com/ubuntu bionic-security InRelease? ? ? ? ? ? ? ?
Ign:6 https://pkg.jenkins-ci.org/debian binary/ Release.gpg
Reading package lists... Done?
W: GPG error: https://pkg.jenkins-ci.org/debian binary/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FCEF32E745F2C3D5
E: The repository 'https://pkg.jenkins-ci.org/debian binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
        

...there is an easy fix for this error. Copy the code after "NO_PUBKEY" from the error log and use it like this -

ubuntu@ip-172-31-19-110:~$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv FCEF32E745F2C3D5

Executing: /tmp/apt-key-gpghome.4rt2s8KDG6/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv FCEF32E745F2C3D5
gpg: key FCEF32E745F2C3D5: public key "Jenkins Project <[email protected]>" imported
gpg: Total number processed: 1
gpg:? ? ? ? ? ? ? ?imported: 1
        

This will fix the error. Now, continue with "sudo apt update -y"

ubuntu@ip-172-31-19-110:~$ sudo apt update -y

Hit:1 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease? ? ? ? ? ??
Hit:3 https://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease? ? ? ? ??
Ign:4 https://pkg.jenkins-ci.org/debian-stable binary/ InRelease? ? ? ? ? ? ? ? ? ? ? ? ??
Get:5 https://pkg.jenkins-ci.org/debian-stable binary/ Release [2044 B]? ? ? ? ? ? ? ??
Get:6 https://pkg.jenkins-ci.org/debian-stable binary/ Release.gpg [833 B]
Hit:7 https://security.ubuntu.com/ubuntu bionic-security InRelease? ? ? ? ? ? ? ??
Get:8 https://pkg.jenkins-ci.org/debian-stable binary/ Packages [18.7 kB]
Fetched 21.6 kB in 0s (49.3 kB/s)? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Reading package lists... Done
Reading state information... Done
31 packages can be upgraded. Run 'apt list --upgradable' to see them.


        

Finally, run "sudo apt install jenkins -y" to install Jenkins

ubuntu@ip-172-31-19-110:~$ sudo apt install jenkins -y

Reading package lists... Done
Building dependency tree? ? ? ?
Reading state information... Done
The following NEW packages will be installed:
? jenkins
0 upgraded, 1 newly installed, 0 to remove and 31 not upgraded.
Need to get 67.0 MB of archives.
After this operation, 67.4 MB of additional disk space will be used.
Get:1 https://pkg.jenkins-ci.org/debian-stable binary/ jenkins 2.249.3 [67.0 MB]
Fetched 67.0 MB in 6s (11.5 MB/s)? ?
Selecting previously unselected package jenkins.
(Reading database ... 73114 files and directories currently installed.)
Preparing to unpack .../jenkins_2.249.3_all.deb ...
Unpacking jenkins (2.249.3) ...
Setting up jenkins (2.249.3) ...
Processing triggers for systemd (237-3ubuntu10.42) ...
Processing triggers for ureadahead (0.100.0-21) ...        

• EDIT: The above command might not work anymore. Here is an alternative:

-Download the jenkins file using this-

wget?https://pkg.jenkins.io/debian-stable/binary/jenkins_2.332.3_all.deb

-and then install it using this, you may require sudo access for the below command to work.

dpkg -i jenkins_2.332.3_all.debe        

• Start Jenkins server

ubuntu@ip-172-31-19-110:~$ sudo systemctl start jenkins

ubuntu@ip-172-31-19-110:~$ sudo systemctl status jenkins

● jenkins.service - LSB: Start Jenkins at boot time
? ?Loaded: loaded (/etc/init.d/jenkins; generated)
? ?Active: active (exited) since Sat 2020-11-21 19:49:23 UTC; 1min 55s ago
? ? ?Docs: man:systemd-sysv-generator(8)
? ? Tasks: 0 (limit: 1140)
? ?CGroup: /system.slice/jenkins.service


Nov 21 19:49:22 ip-172-31-19-110 systemd[1]: Starting LSB: Start Jenkins at boot time...
Nov 21 19:49:22 ip-172-31-19-110 jenkins[22173]: Correct java version found
Nov 21 19:49:22 ip-172-31-19-110 jenkins[22173]:? * Starting Jenkins Automation Server jenkins
Nov 21 19:49:22 ip-172-31-19-110 su[22225]: Successful su for jenkins by root
Nov 21 19:49:22 ip-172-31-19-110 su[22225]: + ??? root:jenkins
Nov 21 19:49:22 ip-172-31-19-110 su[22225]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Nov 21 19:49:22 ip-172-31-19-110 su[22225]: pam_unix(su:session): session closed for user jenkins

Nov 21 19:49:23 ip-172-31-19-110 jenkins[22173]:? ? ...done.
Nov 21 19:49:23 ip-172-31-19-110 systemd[1]: Started LSB: Start Jenkins at boot time.        

• Jenkins runs on port 8080. In order for a user to access the JenkinsUI - that port must be accessible from the internet. To make it so, the port must be opened through ufw which is a netfilter firewall inside ubuntu.

ubuntu@ip-172-31-19-110:~$ sudo ufw status
Status: inactive


ubuntu@ip-172-31-19-110:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? 
Firewall is active and enabled on system startup

ubuntu@ip-172-31-19-110:~$ sudo ufw allow 8080
Rule added
Rule added (v6)

ubuntu@ip-172-31-19-110:~$ sudo ufw allow OpenSSH
Rule added
Rule added (v6)
        

• To access the Jenkins master node, allow inbound access to port 8080 in the Security Group settings of the instance. After that, go to https://<IP of ec2 instance>:8080/ and complete the UI setup by following the instructions there.

Step 2: Installing the EC2 plugin inside Jenkins.

• From the Jenkins UI, go to Manage Jenkins -> Manage Plugins -> Available
• Search for 'Amazon EC2 plugin' and install it. There is no need to restart Jenkins after installing the plugin.

Step 3: Setting up the Plugin to provision worker nodes dynamically.

• From the Jenkins UI, go to Manage Jenkins -> Manage Nodes & Clouds -> Configure Clouds
• Add a new EC2 cloud
• The way the plugin works is that - it requires information about the worker node Jenkins wants to provision. Click on 'Advanced' to see all the required fields.

Details such as the Image, Security Group, Init script, VPC, Subnet for the worker node are required so that the plugin can correctly configure the worker node.

Details such as AWS access keys are required so that the plugin has permissions to rollout an EC2 on its own. Using an IAM user with enough permissions to rollout an EC2 is recommended.

Details such as Connection Strategy, Connect by SSH Process, Host Key Verification Strategy are required so that the plugin knows the method by which the master node will communicate with the worker nodes.

For more details of the plugins options - go through the plugin's documentation.

• Create an IAM user with enough permissions to rollout an EC2 instance i.e. 'AmazonEC2FullAccess' policy.
• Create a key-value pair. Download it.
• Create a Security Group which allows ssh (port 22) inbound rule.
• As the plugin expects the workers to have Java8 installed prior to joining the master node. Java8 should be installed via the 'init script'.

sudo apt update -y
sudo apt install openjdk-8-jdk openjdk-8-jre -y
        

The configurations I used are shown below.

AMI ID: ami-0dd9f0e7df0f0a138 (ubuntu18.04) 
Instance Type: T2Micro
Security group names: jenkins_worker_sg
Remote FS root: /home/ubuntu
Remote user: ubuntu
AMI Type: unix

Labels: aws_ec2
Usage: Only build jobs with label expressions matching this node

Init script: sudo apt update -y
             sudo apt install openjdk-8-jdk openjdk-8-jre -y
Subnet IDs for VPC: subnet-d4beafae
Associate Public IP: Yes
Connection Strategy: Private
Connect by SSH Process: Yes
Host Key Verification Strategy: check-new-soft
Maximum Total Uses: -1
        

• Click on apply

About the connection strategies: The connection strategy dictates how the plugin is going to connect to the newly provisioned worker node to make sure that the new node has been configured correctly. The options are 'check-new-hard', 'check-new-soft','accept-new' and 'off'. Details of these options can be found at https://plugins.jenkins.io/ec2/.

About associating public ips: It is recommended to keep this option enabled, as it will allow the users to ssh into the worker nodes from their local system for debugging purposes.

Step 4: Test the setup.

Create a new Jenkins Job. Under the general configure settings of the job - Choose the 'Restrict where this project can be run' option. Write the label which you used in the setup above. In my case, it was 'aws_ec2'. Finally, start the job.

Be patient!

It is normal for the worker node to take about 7-10 minutes to become ready. Keep a check on the worker node's logs. What is happening behind the scenes is that the EC2 plugin is making requests to the worker node...it will keep doing so until it gets a correct response or a timeout occurs.

If a timeout occurs, you must recheck the configurations of the cloud and try to debug the problem by going through the master node logs. To see the master node logs - go to Jenkins UI -> Manage Jenkins -> System Logs -> All Jenkins Logs

When a worker node joins the master node successfully, the logs would like this-

Step 5 (optional): Optimization

• Notice that each time a worker is provisioned, the init script needs to be run. A much better approach to this would be to create a custom ubuntu18.04 AMI with Java8 pre-installed. And use this AMI instead.
• A heavier instance type than t2.micro like c5.large etc. can be also used to speed up builds. Since the worker node will cease to exist after the Jenkins job is complete - the cost would be minimal. This is one of the advantages of using dynamically provisioned worker nodes.
• Additional requirements such as docker, nodejs, npm libraries should also be installed via an init script or inside the custom AMI.

I hope this article was helpful. Let me know what you think.

Drop a message if you have any queries.