?? Efficient CI/CD Implementation for a Node.js app using Jenkins and Amazon S3 for artifact management

https://github.com/OluwaTossin/my-node-app

Project Overview:

This project aims to set up a Continuous Integration/Continuous Deployment (CI/CD) pipeline for a Node.js application using AWS services. The pipeline automates the build, test, and deployment processes, ensuring smooth and efficient software delivery. The end goal is to deploy a sample Node.js application using a CI/CD pipeline managed by Jenkins, integrated with GitHub for version control, and AWS S3 for artifact management.

Step 1: Set Up AWS Account (if you do not already have one)

Create and configure an AWS account.

Sign up for an AWS account. Set up IAM roles and policies for Jenkins and other necessary services.

Step 2: Prepare the Node.js Application

1. Install Node.js and npm (Node Package Manager)

Download and install Node.js from the official Node.js website .

This will also install npm, which is used to manage packages for Node.js.

verify they are installed:

a. Create a Project Directory

Open your terminal (Command Prompt, PowerShell, or Terminal on macOS/Linux).

Create a new directory for your project and navigate into it:

mkdir my-node-app        

cd my-node-app        

b. Initialize the Node.js Project

Run the following command to create a package.json file:

npm init -y        

This command initializes a new Node.js project with default settings. The package.json file manages your project's dependencies and scripts.

2. Write a Simple "Hello World" Application

a. Create the Application File

Create a new file named app.js in your project directory named: app.js

Write the "Hello World" Code:

const http = require('http');

const hostname = '127.0.0.1';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello, World!\n');
});

server.listen(port, hostname, () => {
  console.log(`Server running at https://${hostname}:${port}/`);
});        

This code creates a basic HTTP server that responds with "Hello, World!" when accessed.

C. Run the Application

In your terminal, run the application using Node.js:

node app.js        

Open your web browser and go to https://127.0.0.1:3000/ to see your "Hello, World!" message.

Step 3: Set Up GitHub Repository

a. Create a GitHub Repository

In your GitHub account, click the "New" button to create a new repository. Name your repository (e.g., my-node-app), add a description, and click "Create repository".

b. Initialize Git and Commit Your Code

In your terminal, initialize a Git repository in your project directory:

git init        

git add .        

git commit -m "Initial commit"        

C. Push Your Code to GitHub

Link your local repository to the GitHub repository you created:

git remote add origin https://github.com/your-username/my-node-app.git        

git push -u origin main        

The Node.js application is now prepared, and the code is committed to GitHub. This setup forms the foundation for building and deploying your application using a CI/CD pipeline.

Step 4: Set Up Jenkins on AWS

a. Launch an EC2 Instance.

Give your instance a name (e.g., "Jenkins-Server"). Select "Ubuntu Server 20.04 LTS (HVM), SSD Volume Type" (AMI). Select the "t2.micro" instance type, which is eligible for the Free Tier. Launch Instance. Connect to your instance using SSH.

b. Install Jenkins on the EC2 Instance

1. Update the Package Index:

Once connected to your instance, run the following commands to update the package index:

sudo apt update        

sudo apt upgrade -y        

c. Install Java:

Jenkins requires Java to run. Install OpenJDK 11:

sudo apt install openjdk-11-jdk -y        

d. Add the Jenkins Repository and Key:

Add the Jenkins repository key to the system:

curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null        

?Add the Jenkins package repository:

echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null        

e. Install Jenkins:

Update the package index and install Jenkins:

sudo apt update        

sudo apt install jenkins -y        

f. Start and Enable Jenkins Service:

Start the Jenkins service:

sudo systemctl start Jenkins        

Enable Jenkins to start on boot:

sudo systemctl enable Jenkins        

?Access Jenkins:

Open your web browser and navigate to https://your-ec2-instance-public-dns:8080.You should see the Jenkins setup screen.

Configure Jenkins with Necessary Plugins

1. Unlock Jenkins:

To unlock Jenkins, you need the initial admin password. Run the following command to retrieve it:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword        

Copy the password and paste it into the Jenkins setup screen.

G. Install Suggested Plugins:

On the setup screen, choose "Install suggested plugins". Jenkins will automatically install the recommended plugins.

Create an Admin User: After the plugins are installed, you will be prompted to create an admin user. Fill out the required information and complete the setup.

h. Install Additional Plugins:

Once you are logged in, navigate to Manage Jenkins > Manage Plugins > Available. Search for and install the following plugins:

·???????? GitHub Integration

·???????? AWS Credentials

I. Configure Global Tool Configuration:

Go to Manage Jenkins > Global Tool Configuration. Configure the necessary tools, such as Git.

The Jenkins server is now set up and ready to use. You can start creating Jenkins jobs and configuring your CI/CD pipeline.

Step 5: Integrate Jenkins with GitHub

A. Link Jenkins with the initially created GIT repo.

1. Create a New Jenkins Job:

Go to your Jenkins dashboard. Click on New Item on the left-hand menu.

1. Name and Select Job Type:

Enter a name for your job (e.g., MyNodeAppBuild). Select Freestyle project. Click OK.

1. Configure Source Code Management:

In the job configuration page, scroll down to Source Code Management. Select Git.

Enter your GitHub repository URL https://github.com/OluwaTossin/my-node-app

1. Add Credentials:

If your repository is private, click Add next to Credentials. Choose Username with password and enter your GitHub username and personal access token. Select the credentials you just added from the dropdown.

1. Configure Branches to Build:

In the Branches to build section, leave it as */main (or adjust it to match your branch name).

6. Build Triggers:

Scroll down to the Build Triggers section. Check GitHub hook trigger for GITScm polling.

7. Add Build Steps:

Scroll down to the Build section. Click Add build step. Select Execute shell.

Enter the commands to build your Node.js application:

npm install 
npm test        

B. Configure Webhook in GitHub to Trigger Jenkins Builds on Code Commits

1. Go to Your GitHub Repository:

Navigate to your GitHub repository

1. Access Repository Settings:

Click on the Settings tab.

1. Configure Webhooks:

In the left-hand menu, click Webhooks. Click Add webhook.

1. Set Up Webhook:

Payload URL: Enter your Jenkins server URL followed by /github-webhook/. For example: https://your-jenkins-server/github-webhook/ .

Content type: Select application/json.

Secret: Leave it blank unless you want to use a secret for added security (then you need to configure Jenkins accordingly).

Which events would you like to trigger this webhook?: Select Just the push event.

Click Add webhook.

Testing the Integration

1. Push a Change to GitHub:

Make a change in your repository and push it to the main branch. This should trigger the webhook and start a build in Jenkins.

1. Check Jenkins:

Go to your Jenkins dashboard. You should see a new build started for your job (MyNodeAppBuild).

Step 6: Set Up Build Tools and Scripts

A. Write a Build Script to Automate the Build Process

1. Create a Build Script:

Open your project in VSCode or any text editor. Create a new file named build.sh in your project root directory.

1. Write the Build Script:

Add the necessary commands to automate the build process. For a Node.js application, it typically includes installing dependencies and running tests.

Example build.sh :

#!/bin/bash

# Exit immediately if a command exits with a non-zero status
set -e
echo "Installing dependencies..."
npm install
echo "Running tests..."
npm test
echo "Build complete."        

Make the Script Executable:

Ensure the script has executable permissions. Run the following command in your terminal:

chmod +x build.sh        

B. Configure Jenkins to Use the Build Script

1. Access Jenkins Dashboard and edit Jenkins Job: Go to your job (MyNodeAppBuild).

Click Configure.

1. Add Build Step:

Scroll down to the Build section. Click Add build step. Select Execute shell.

1. Use the Build Script:

In the Command box, enter the path to your build script:

#!/bin/bash

set -e
echo "Installing dependencies..."
npm install
echo "Running tests..."
npm test
echo "Build complete."        

Save the Job Configuration. Click Save to apply the changes.

4. Install Node.js on Jenkins Server

sudo apt update        

sudo apt install -y nodejs npm        

Verify Installation:

node -v        

npm -v        

Testing the Setup

1. Commit and Push build.sh : Ensure build.sh is in your repository.

git add build.sh 
git commit -m "Add build script" 
git push origin main        

Go to your Jenkins job. Click Build Now.

Check Console Output: Verify that the build script runs correctly and the build completes successfully.

Step 7: Implement Continuous Integration

This stage is to set up automated testing in the CI/CD pipeline.

A. Write Unit Tests for the Node.js Application

1. Install a Testing Framework:

Install Mocha and Chai, which are popular testing frameworks for Node.js.

npm install --save-dev mocha chai        

2.????? Create Test Directory and Files:

Create a tests directory in your project root. Add a test file, e.g., tests/app.test.js.

3.????? Write a Simple Test:

Here is an example test using Mocha and Chai:

// tests/app.test.js
import { expect } from 'chai';
describe('Simple Test', () => {
  it('should return true', () => {
    expect(true).to.be.true;
  });
});        

B. Update package.json to Use Mocha:

{

  "name": "my-node-app",

  "version": "1.0.0",

  "main": "index.js",

  "type": "module",

  "scripts": {

    "test": "mocha tests/*.test.js"

  },

  "keywords": [],

  "author": "",

  "license": "ISC",

  "description": "",

  "devDependencies": {

    "chai": "^5.1.1",

    "mocha": "^10.4.0"

  }

}        

Commit and Push Changes:

git add . 
git commit -m "Update to ES Modules" 
git push origin main        

C. Configure Jenkins to Run Tests

1. Edit Jenkins Job Configuration:

Go to your Jenkins job configuration. Ensure the build step includes running the tests.

1. Example Script for Jenkins:

Update the script in Jenkins to run tests as part of the build:

#!/bin/bash
set -e
echo "Installing dependencies..."
npm install
echo "Running tests..."
npm test
echo "Build complete."        

Step 8: Set Up Artifactory (Using AWS S3)

Here, we will configure S3 for artifact management and integrate it with Jenkins.

1. Set Up an S3 Bucket

Go to the AWS Management Console. Go to the S3 service. Click on Create bucket. Enter the Bucket name as my-node-app-artifacts. Choose a Region. Click on Create bucket.

1. Configure Jenkins to Publish Build Artifacts to S3

Install AWS CLI on the Jenkins Server

SSH into your Jenkins server. Run the following commands to install AWS CLI:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"        

unzip awscliv2.zip        

sudo ./aws/install        

3.????? Verify the Installation:

Run:

aws –version        

You should see the AWS CLI version printed in the terminal.

Configure AWS Credentials in Jenkins

1. Create AWS Credentials:

In Jenkins, go to Manage Jenkins > Manage Credentials. Select the appropriate domain (e.g., Global). Click on Add Credentials. Choose AWS Credentials as the kind. Enter your AWS Access Key ID and Secret Access Key.

Provide an ID and description for easy identification (e.g., ID: aws-credentials, Description: AWS Credentials for S3). Save the credentials.

Create a Jenkins Pipeline

1. Create a New Pipeline Job:

Go to Jenkins and click on New Item. Enter the job name (e.g., Pipeline). Select Pipeline and click OK.

1. Configure the Pipeline Script:

In the Pipeline section, select Pipeline script. Use the following script:

pipeline {
    agent any
    environment {
        AWS_ACCESS_KEY_ID = credentials('aws-credentials')
        AWS_SECRET_ACCESS_KEY = credentials('aws-credentials')
    }
    stages {
        stage('Cleanup') {
            steps {
                script {
                    // Clean up the workspace
                    deleteDir()
                }
            }
        }
        stage('Checkout') {
            steps {
                script {
                    // Clone the repository
                    sh 'git clone https://github.com/OluwaTossin/my-node-app.git'
                }
            }
        }
        stage('Build') {
            steps {
                script {
                    // Navigate into the cloned repository directory
                    dir('my-node-app') {
                        // Install dependencies and run tests
                        sh 'npm install'
                        sh 'npm test'
                        // Package the application
                        sh 'npm pack'
                    }
                }
            }
        }
        stage('Publish to S3') {
            steps {
                script {
                    // Navigate into the cloned repository directory
                    dir('my-node-app') {
                        // Upload artifacts to S3
                        sh '''
                        aws s3 cp my-node-app-1.0.0.tgz s3://my-node-app-artifacts/builds/${BUILD_NUMBER}/
                        '''
                    }
                }
            }
        }
    }
}        

Save and Run the Pipeline:

Save the pipeline configuration. Click on Build Now to run the pipeline.

Step 9: Conduct Regular Tests and Maintenance

This stage is designed to ensure the reliability and stability of the CI/CD pipeline.

1. Schedule Regular Builds and Tests in Jenkins

Step-by-Step Breakdown:

1. Access Jenkins Dashboard: Find and click on the job you want to schedule regular builds for. Click on "Configure" on the left-hand side of the job page. Scroll down to the "Build Triggers" section. Select the checkbox for "Build periodically."
2. Specify the Build Schedule:

Enter the schedule using cron syntax in the "Schedule" field.

1. Example to run the job every 15 minutes: H/15
2. Example to run the job daily at midnight: 0 0 *

3. Save the Configuration:

Scroll down and click the "Save" button.

Cron Syntax Guide:

• * (Every minute)
• H/15 (Every 15 minutes)
• 0 0 * (Daily at midnight)
• 0 12 0 (Every Sunday at noon)

2. Monitor the Pipeline for Issues and Resolve Them Promptly

Access Jenkins Dashboard: Monitor Job Status: Check the status of your jobs on the dashboard. Look for any failed builds indicated by red status.

View Build Logs:

Click on the job name and then click on the specific build number that failed.

Click on "Console Output" to view the build logs.

Identify Issues (if any):

Read through the logs to identify any errors or issues that caused the build to fail.

Fix Issues:

Make necessary changes to your code or pipeline configuration to fix the identified issues.

Commit and push the changes to your repository.

Rebuild the Job:

Trigger a new build manually by clicking on "Build Now" on the job page.

Verify that the build passes successfully.

Set Up Notifications:

Configure Jenkins to send email or Slack notifications for failed builds.

Go to "Manage Jenkins" > "Configure System" and set up the "Extended E-mail Notification" or "Slack Notification."

3. Update Dependencies and Scripts as Needed to Maintain the Pipeline

Review Dependencies:

Open your project repository in your code editor (e.g., Visual Studio Code). Open the package.json file to review the dependencies.

Update Dependencies:

Use the terminal to update dependencies to their latest versions. Run npm update to update packages listed in package.json. Alternatively, use specific commands to update individual packages:

Example: npm install <package-name>@latest

Test Updates Locally:

Run your tests locally to ensure that updating dependencies does not break your application. Use npm test to run your tests.

Commit and Push Changes:

Commit the updated package.json and package-lock.json files to your repository. Push the changes to your version control system (e.g., GitHub).

Update Build and Deployment Scripts:

Review and update your Jenkins pipeline script (Jenkinsfile) if necessary. Make sure any changes in dependencies or project structure are reflected in the build and deployment steps.

Run a New Build:

Trigger a new build in Jenkins to ensure that the updated dependencies and scripts work correctly. Verify that the build and deployment are successful.

Conclusion

This project successfully established a robust CI/CD pipeline for a Node.js application using Jenkins and AWS services. Through the integration of Jenkins with GitHub, I have ensured that the application code was continuously built, tested, and packaged. Key achievements include:

• Automated Builds and Testing: Configured Jenkins to monitor the GitHub repository, trigger builds on code commits, and run automated tests using Mocha and Chai.
• Artifact Management with S3: Utilized Amazon S3 for managing build artifacts, simplifying the storage and retrieval process.
• Continuous Integration: Implemented a streamlined process for continuous integration, ensuring code quality and functionality are maintained with each update.
• Regular Maintenance: Scheduled regular builds and tests in Jenkins, with a strategy in place for monitoring and promptly addressing any pipeline issues.

This project demonstrates essential DevOps practices, leveraging Jenkins and AWS to create an efficient and reliable CI/CD pipeline. This project highlights the practical application of CI/CD principles and the effective use of cloud services to streamline software development processes.