Web components CI/CD with Jenkins, dynamic agents and Docker

Introduction

The company that I currently work for as a DevOps Engineering Manager is amongst the ones at the fore-front of technology and innovation. Most of you may know it as the company responsible for the most famous designer tools used by both individual creators and by enterprises alike.

You would be surprised to find out that actually many of the functionalities included in the products we ship are written using the world’s most popular scripting languages: JavaScript. Besides JavaScript usage,? the toolchain used by the various developments teams which are working on bringing functionality at the highest quality and speed to you makes use of a modern tech stack and its versions:

·?????? TypeScript

·?????? WebPack 5

·?????? Bazel

·?????? React etc.

As the company is quite large, we have multiple, smaller teams working on different product components as well as larger teams working on a single product. This possess unique challenges when talking about how can we make sure that the quality of our code is exceptional and that little to no bugs "escape" to the production environment.

In a nutshell, the architecture of our products is very similar to a FOA (Feature Oriented Architecture) where each F (feature) is a JavaScript component itself and is included inside the host application via some custom, in-house bred mechanisms.

This article aims to exemplify how the team I am leading has standardised the way component development teams build and deliver their?features/components so that we lessen the usage of FOA (Feature Oriented Architecture).

From the start, note that the process that we'll pe depicting in the upcoming sections refers to a subset of components and teams which are using a similar tech stack (JS and / or TS with or without React)? however having different build systems and scattered build logic and, as a consequence, various issues caused by underlying infrastructure and its architecture.

The goal of these teams though is the same: to be able to swiftly build and deliver components to host applications (native or web based) which can consume them by either static imports or at runtime by using them as WebPack federated modules (for web).

The WebPack federated modules are loaded inside our web products via an in-house built loader SDK which simplifies and standardises how web components can be retrieved within our next generation web products whilst for the desktop and MacOS applications loading is done at compile time but also at runtime by using proprietary technologies.

Existing Challenges

Most problems the teams encountered with their CI / CD infrastructure was due to build agents (which were actually physical machines) getting clogged up and needing periodically storage cleaning. Due to the fact the teams were mostly comprised of Frontend and/or full-stack engineers and experience on build systems and infrastructure was not their forte, a single team was created to cater and support a new (to some extent) build system - the team I am currently leading.

For historical reasons, let's go over the high level existing infrastructure and CI and CD toolchain:

• Github Enterprise for source code versioning and control
• Jenkins for continuous integration jobs like unit testing, security scanning the third party libraries and building the components
• GoCD for continuous delivery jobs like deploying the WebPack modules?
• AWS for storing (S3) the federated modules / web components
• Akamai for delivering the web components to the host products?

?

Build System Requirements

Part of the effort to simplify and to standardise the process of CI / CD for the component development teams, a new initiative was born.

In the following, we will go through the process that the team that I lead (which is, again, a DevOps team) has undertaken in order to simplify, standardise and reduce maintenance costs for the component delivery pipeline.

The main goals of the component development teams were simple:

1. have a new build system that is familiar to them but that doesn't have all the drawbacks of the current one, more accurate, get rid of the infamous Jenkins agents being out of storage space.
2. enable the developers to manually merge a PR into the default branch. Do not auto-merge
3. standardise (as much as possible) the CI and CD processes across the feature / component teams so that cross team collaboration is lessened
4. use a single system for both CI and CD
5. accommodate the transition with minimal to no changes to the product / component repositories
6. provide fast PR validation and if possible, add the functionality to have PR build artefacts at each PR commit
7. have a leaner Jenkinsfile (no DSL) and no freestyle jobs

Not that many requirements for starters but we’ll get to that later.

Technical Implementation

Given that we needed to standardise he CI/CD processes for the web components development teams as much as possible but in the same time cater for each of their specificity and support multiple operating systems in a non complicated architecture, we chose Jenkins over the other tools we have researched. We have looked into using Argo Workflows and Argo CD, Circle CI, Github Actions (which we actually couldn't use on our hosted Github Enterprise feature set) however none of the ones which we could use were simple and could easily fit the purpose of the development teams.

In order to reduce the number of dependencies and architectural overlays, we decided to give up using Jenkins atop a Kubernetes cluster and instead choose dynamically spawned nodes in AWS EC2 by using this plugin. One motivation atop others made us take this decision: we also needed to run UI tests on OS's like Windows and MacOS.

We have also gone down the road of creating a Jenkins Shared Library which makes use of dynamically spawned EC2 nodes and Docker containers to run all pipeline stages.? For the future, we are considering of switching to using JTE (Jenkins Templating Engine) in order to provide a more composable approach for building custom build pipelines by the development teams.

The section below snapshots the high level architecture of our Jenkins infrastructure and Shared Library.

Architecture

Infrastructure diagram

As already mentioned we wanted to keep things simple from 2 perspectives:

1. we should have an easy means to develop and debug potential issues?
2. we should keep costs and complexity at a minimum

Between the two options presented below, we chose the one of the left as it allowed us the following:

1. reduce complexity of the infrastructure by eliminating the K8s pods. In our current examples, the containers are running directly on the dynamically spawned nodes.?
2. transfers the auto-scaling responsibility to the plugin whilst we simply define the required thresholds (number of nodes to be spawned, container requirements / node etc)
3. reduces maintenance and development and debugging . No more managing pod specs. No associated costs with managing EKS or another Kubernetes cluster just for building some artefacts which would be delivered from CDN

?

?

Code base diagram

When developing Jenkins Shared libraries, we are advised to use Groovy. So we did. The main toolchain libraries and SDK's that we are using are? :

·?????? Groovy 4.0.6

·?????? Gradle 6.8

·?????? Codenarc (for linting)

·?????? Kohsuke Github library - for any operation concerning Github repos (checking PR labels, branches etc.)

·?????? JUnit, Mockito and Jenkins unit testing library for unit testing

The Jenkins Shared library tutorials (available here: https://www.jenkins.io/doc/book/pipeline/shared-libraries/ ) instruct you that whatever commonly used functionality you need to build you should add in the vars folder.

On our end that would have involved in a lot of files inside the ‘vars’ folder so we decided to go all in OOP as it makes unit testing much easier.

This is the high level code base architecture. Please note that not all stages or wrappers or any sensitive / specific classes were added but it should give you a birds eye view on how we structured our codebase

Pipeline entrypoints

Currently we support 6 entry points (Groovy scripts) in our Shared Library as following:

1. component.groovy → used for building, publishing (in Artifactory) and deploying(on CDN) the components
2. cdnDeploy.groovy → used by the development teams to publish built artefacts to the production environment (CDN) at any time
3. appPackage.groovy → used for building and publishing npm packages in Artifactory. This is very similar to the component.groovy but much lighter
4. smokeTests.groovy → used by the development teams to run smoke tests on artefacts already deployed to CDN?
5. webApp.groovy → used for building and deploying web applications
6. namedReleases.groovy → used for building aliased CDN paths for retrieving web components (no version number in the URL)

Each entry point is very slim and every bit of business logic lies inside the underlying classes

#!/usr/bin/env groovy
import com.company.ComponentIntegrationAndDelivery

void call(Map<String, ?> options = null ) {
    ComponentIntegrationAndDelivery cicd = new ComponentIntegrationAndDelivery(this, options)
    cicd.run()
}        

Custom Wrappers

To ease up creating and using? standard closures on Jenkins objects like container, node, stage etc, we have created our own wrappers like the one exemplifying the Container below

class Container {

    Object pipelineScript
    DockerProvider dockerProvider

    Container(Object script) {
        this.pipelineScript = script
        this.dockerProvider = new DockerProvider(script)
    }

    void wrap(ImageNames image, PipelineContext pipelineContext, List<String> envVars = [], Closure closure) {
        String dockerImage = this.dockerProvider.getDockerImages(image, pipelineContext)[0]
        this.pipelineScript.docker.withRegistry(this.dockerProvider.dockerReadRegistry, this.dockerProvider.getRegistryCredentialId()) {

       this.pipelineScript.docker.image(dockerImage).pull()
            this.pipelineScript.docker.image(dockerImage).inside("${pipelineContext.containerResources} ${envVars.join(' ')}") {
                closure()
            }
        }
    }
}
        

?

PipelineObject

The PipelineObject is the abstract class which serves as the parent for all stages since it provides the Jenkins script Object that helps us execute commands in the containers. Here’s an excerpt of it:

abstract class PipelineObject {

    Object pipelineScript
    Logger logger


    PipelineObject(Object pipelineScript, PipelineContext pipelineContext) {
    	this.pipelineScript = pipelineScript
    	this.pipelineContext = pipelineContext
    	this.logger = pipelineContext.getLogger()
    }
}?        

Any object extending this abstract class will have the base requirements for running any Jenkins commands since it has access to the pipelineScript which is the Jenkins script context.

PipelineContext and JobOptions

We have tens of pipeline runs per minute and each pipeline runs with their own build options.

These buildOptions are nothing more than Map entries (K,V pairs) but at runtime they are extracted into something that we call:

PipelineJobOptions:

• containing the characteristics of the pipeline

PipelineContext:

• which contains all the PipelineJobOptions and abstracts them from the consumer AND
• all the necessary methods for adding for that specific pipeline type (whether needed for a component building and deployment or a web application building and deployment script)

class PipelineContext extends PipelineObject {

    private PipelineJobOptions jobOptions

    private GithubUtils ghUtils
    private GeneralUtils generalUtils
    private NodeUtils nodeUtils
    private SlackNotifier slackNotifier
    private Stasher stasher
    private Tracker tracker
    private GeneralWrappers wrappers

    PipelineContext(Object script) {
        super(script)
    }

    PipelineContext init(Map<String, ?> buildOptions, ProjectTypes projectType) {
        
        jobOptions = new PipelineJobOptions()
        initializePipelineJobContext(buildOptions, projectType)

        return this
    }
        

Stages

For easy development , debugging and development, we made sure that each build stage benefits of its own class. Each stage class extends the PipelineObject thus making sure it has all it needs to run other Jenkins commands.

Also, each class expects to be provided with the PipelineContext object.

class CheckoutStage extends PipelineObject {

    CheckoutStage(Object script, PipelineContext pipelineContext) {
        super(script, pipelineContext)
    }

    void checkout(boolean showStage = true) {
        GithubUtils ghUtils = pipelineContext.getGhUtils()
        GeneralWrappers wrappers = pipelineContext.getGeneralWrappers()

        wrappers.stage(StageNames.CHECKOUT, showStage) {
          // ... does checkout 
        }
    }

}?        

Jenkinsfile

One of the most important requirements from the development teams but also aspirations for us, as a DevOps team was to simplify the usage of the CI/CD? infrastructure and process.

If before the introduction of the Jenkins shared library each team had their own flavour of pipeline: DSL based, freestyle in Jenkins, freestyle with a lot of scripts in their product repository, we consolidated it all into a single library and provided a very simple Jenkinsfile structure.

The Jenkinsfile now contains a contractual based Map<String, ?> which instructs the pipeline how to behave and what do to. It’s definition is very easy and simple to understand as defined below.

?

@Library("shared-library@release_v1") _

def buildOptions = [
    'projectName': '{your_project_name}',
    'hasUnitTests': true,
    'runInParallel': true,
    'hasE2e': false,
    'hasWebAppExample': false,
    'nodeVersion': 20
]

component(buildOptions)
        

?

End to end flow

We have discussed how the pipeline has been implemented at a high level now let's see how the end-2-end flow looks like.

In case of PR builds, before the Artifactory publish stage we have on more stage that validates the PR and sets a specific status check in Github which enables the Merge button in Github to be active and clicked. This was one extra requirement from the development teams - that the PR's shouldn't be merged automatically by the build system rather the developers should have the latest saying in what concerns the merges.

Also, the same stage mentioned above as well as the CDN deploy stage are optional in case of PR's and they can either be acted upon manually or automatically by the pipeline by adding a label on the PR, from Github.

After all checks have passed for the default branch and the artefacts have been published (in Artifactory and CDN or just in the first one), the host applications can import them freely either al compile time via their static imports list or at runtime via the in-house built component loader SDK.

Conclusions

Choosing the non Kubernetes approach allowed us to have reduced costs as we do not have to maintain a fully fledged K8s cluster and gives us the flexibility to also have dynamically and short-lived spawned agents (Linux and Windows in EC2).

The disadvantages (if you can consider them so) are that for MacOS we have reached the conclusion that it's actually cheaper to buy our own, physical machines compared to renting them from AWS due to high costs.