CI/CD Pipeline for Scalable Web App using AWS Amplify, AWS CodeBuild, AWS CodePipeline, Terraform, Lambda, and CloudWatch for Monitoring

Project Repo: https://github.com/OluwaTossin/aws-devops-project????

Objectives:

In this project, I will design and implement an AWS infrastructure to support a highly available and scalable web application, consisting of a Next.js frontend hosted on AWS Amplify, a Node.js backend hosted on AWS Lambda, and a MySQL database hosted on Amazon RDS. I will use Terraform for infrastructure as code (IaC) to provision the AWS resources. Additionally, I will set up a CI/CD pipeline for automated deployment using AWS CodePipeline, CodeBuild, and AWS Amplify. I will implement monitoring and alarms using AWS CloudWatch to monitor the application's performance and health.

The key AWS services and tools used in this project are:

• Terraform for infrastructure provisioning
• AWS Lambda for hosting the backend Node.js services
• AWS Amplify for deploying the React/Next.js frontend
• Amazon RDS for hosting the MySQL database
• AWS CodePipeline and CodeBuild for setting up the CI/CD pipeline
• AWS CloudWatch for monitoring and logging

Step 1: Set Up GitHub Repository

1. Create a GitHub Repository

1. Go to GitHub and sign in.
2. Click on the + icon in the top-right corner and select New repository.
3. Name the repository, aws-devops-project.
4. Set the repository to Public or Private.
5. Add a README file.
6. Click on Create repository.

2. Clone the Repository Locally

To clone the repository to your local machine, follow these steps:

1. Open your terminal or command prompt.
2. Run the following commands:

git clone https://github.com/your-username/aws-devops-project.git
cd aws-devops-project        

Step 2: Set Up The Development Environment

To begin development, install the necessary tools:

1. Node.js and npm

Node.js is a JavaScript runtime that allows you to run JavaScript on the server side. npm is the package manager for Node.js.

Installation Steps:

1. Go to the Node.js download page.
2. Download and install the LTS version.
3. Verify the installation by running the following commands in your terminal:

node -v
npm -v        

2. AWS CLI

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services.

Installation Steps:

1. Follow the instructions in the AWS CLI installation guide.
2. Verify the installation by running:

aws –version        

3. Terraform

Terraform is an open-source infrastructure as code software tool.

Installation Steps:

1. Go to the Terraform download page.
2. Follow the instructions to install Terraform for your operating system.
3. Verify the installation by running:

terraform -v        

Step 3: Infrastructure Design and Provisioning

I'll use Terraform to provision the infrastructure using the following steps:

1. Initialize Terraform Configuration: Create a directory structure:

mkdir infrastructure
cd infrastructure        

2. ???? Create main.tf:

provider "aws" {
  region = "us-east-1"
}
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}
resource "aws_subnet" "public_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a"
}
resource "aws_subnet" "private_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-1a"
}
resource "aws_subnet" "public_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.3.0/24"
  availability_zone = "us-east-1b"
}
resource "aws_subnet" "private_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.4.0/24"
  availability_zone = "us-east-1b"
}
resource "aws_security_group" "rds_sg" {
  vpc_id = aws_vpc.main.id
  ingress {
    from_port   = 3306
    to_port     = 3306
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
resource "aws_db_subnet_group" "main" {
  name       = "main"
  subnet_ids = [aws_subnet.public_a.id, aws_subnet.private_a.id, aws_subnet.public_b.id, aws_subnet.private_b.id]
}
resource "aws_db_instance" "mysql" {
  allocated_storage    = 20
  engine               = "mysql"
  instance_class       = "db.t3.micro"
  engine_version       = "5.7"
  db_name              = "mydb"
  username             = "admin"
  password             = "password"
  vpc_security_group_ids = [aws_security_group.rds_sg.id]
  db_subnet_group_name = aws_db_subnet_group.main.name
  skip_final_snapshot  = true
}
output "vpc_id" {
  value = aws_vpc.main.id
}        

3.????? Initialize and Apply Terraform:

terraform init
terraform apply        

4. Verify Resources in AWS Management Console: Check the RDS and VPC dashboards to ensure resources are created.

5. Push the Terraform setup to GitHub:

git add .
git commit -m "Add initial Terraform configuration"
git push origin main        

Step 4: Set Up AWS Amplify for Frontend

1. Create a Next.js Application:

Create a new Next.js application using the following command:

npx create-next-app@latest frontend
cd frontend        

2. Start the Development Server Locally

After creating the Next.js application, start the development server to test it locally.

npm run dev        

Open your browser and navigate to https://localhost:3000 to see your Next.js app running locally.

You should see the "Welcome to My Next.js App!" screen as shown in the attached image.

3. Push the Frontend Setup to GitHub:

Navigate to your project directory and push the initial commit to GitHub:

git add .
git commit -m "Initial commit for frontend application"
git push origin main        

3. Set Up AWS Amplify:

a. Go to AWS Amplify:

In the AWS Management Console, type "Amplify" in the search bar at the top and select Amplify from the search results to go to the AWS Amplify dashboard.

b. Create a New App:

Click on Get Started under Deploy.

c. Connect to Your GitHub Repository:

1. Select GitHub as the repository service and click Continue.
2. If prompted, authorize AWS Amplify to access your GitHub account.
3. Select your GitHub repository (aws-devops-project) from the list.
4. Select the branch you want to deploy (usually main).

d. Configure Build Settings:

Amplify will automatically detect the settings for a Next.js project. Review the build settings. If needed, you can customize them by clicking on Edit next to the detected settings. Click Next to proceed.

e. Save and Deploy:

Review your settings in the Review step. Click Save and Deploy. AWS Amplify will start the build and deployment process. This will take a few minutes.

4. Fixing the Build Error:

This build will fail at this point because the build process couldn't find the package.json file, which is essential for running npm run build. This means that AWS Amplify is trying to build from the root directory of your project instead of the frontend directory where your Next.js app is located.

a. Adjust Amplify Build Settings:

Go back to the aws-devops-project project dashboard, click on App settings in the left sidebar. Select Build settings and edit.

b. Modify the buildspec.yml file:

Specify the correct working directory for the build process. Here is the updated amplify.yml:

version: 1.0

frontend:

  phases:

    preBuild:

      commands:

        - cd frontend

        - npm install

    build:

      commands:

        - npm run build

  artifacts:

    baseDirectory: frontend/.next

    files:

      - '**/*'

  cache:

    paths:

      - frontend/node_modules/**/*        

c. Save and Re-run the Build:

Click Save to update the build settings, and re-run the build.

Step 5: Set Up AWS Lambda for Backend

1. Create a Node.js Application:

1. Create a directory for the backend and initialize a Node.js application:

mkdir backend
cd backend
npm init -y        

npm install --save aws-sdk        

Create a Lambda function (index.js):

exports.handler = async (event) => {
    return {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
};        

2. Push the Backend Setup to GitHub:

1. Add, commit, and push the backend setup to GitHub:

git add .
git commit -m "Initial commit for backend application"
git push origin main        

3. Set Up CI/CD for Lambda:

a. Create an AWS CodePipeline:

1. Navigate to AWS CodePipeline in the AWS Management Console: Open the AWS CodePipeline Console. Click on Create pipeline. Enter a name for your pipeline (e.g., LambdaBackendPipeline). Choose a service role or create a new one. Click Next.

b. Set Up Source Stage:

1. Add a Source Stage: Select GitHub (version 2) as the source provider. Connect to your GitHub account and select the repository (aws-devops-project). Choose the branch to trigger the pipeline (main). Click Next.

c. Set Up Build Stage:

1. Add a Build Stage: Select AWS CodeBuild as the build provider. Click Create project to set up a new CodeBuild project. Enter a name for the project (LambdaBuildProject).

?2. Configure the Build Environment: Choose Managed image. Select the operating system and runtime (e.g., Ubuntu, Node.js). Specify the build specification (buildspec file) if you have one, or enter the build commands directly.

3. Create buildspec.yml for the Node.js Lambda Function: Create a buildspec.yml file in the backend directory with the following content:

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 14
    commands:
      - cd backend
      - npm install
  build:
    commands:
      - npm run build
      - zip -r function.zip .
      - aws s3 cp function.zip s3://my-lambda-functions-bucket-001/function.zip
      - aws lambda update-function-code --function-name MyLambdaFunction --s3-bucket my-lambda-functions-bucket-001 --s3-key function.zip
artifacts:
  files:
    - function.zip
cache:
  paths:
    - backend/node_modules/**/*        

?Push buildspec.yml to GitHub:

git add buildspec.yml
git commit -m "Add buildspec for CodeBuild"
git push origin main        

d. Return to CodePipeline and Complete the Build Stage Setup:

1. Select your new CodeBuild project. Click Next.

e. Skip the Deploy Stage:

1. On the "Add deploy stage" screen, click Skip deploy stage. Click Next to proceed.

4. Finalize the Pipeline:

1. Review and Create the Pipeline: Review all stages in the pipeline. Click Create pipeline.
2. Monitor the Pipeline: Ensure that the CodePipeline triggers and deploys the updated Lambda function as expected.

Step 6: Monitoring and Logging

1. Create CloudWatch Alarms for AWS Lambda Invocation Errors and Latency

Open the CloudWatch Console.

Create Alarm for Lambda Invocation Errors

1. In the left-hand navigation pane, click on Alarms.
2. Click Create Alarm.
3. Click Select metric.
4. In the search bar, type Lambda and select Per-Function Metrics.
5. Choose the function MyLambdaFunction.
6. Select the metric Errors.
7. Click Select metric.

8. Configure the alarm details:

Name: LambdaInvocationErrorsAlarm

Condition: Greater than or equal to 1

Period: 1 minute

9. Click Next.

10. Configure actions (e.g., send a notification to an SNS topic).

11. Click Next and review the settings.

12. Click Create Alarm.

Create Alarm for Lambda Latency

1. Repeat the steps to create a new alarm.
2. This time, select the metric Duration.
3. Configure the alarm details: Name: LambdaLatencyAlarm Condition: Greater than or equal to 1000 milliseconds (adjust as needed) Period: 1 minute

4. Click Next and configure actions.

5. Click Next and review the settings.

6. Click Create Alarm.

Enable CloudWatch Logging for AWS Lambda

a. Go to the Lambda Console

1. Open the Lambda Console.
2. Select Your Function.
3. Choose MyLambdaFunction.

b. Enable Logging

1. Scroll down to the Monitoring section.
2. Click on View logs in CloudWatch.

Enable CloudWatch Logging for AWS Lambda

a. Go to the Lambda Console

1. Open the Lambda Console.
2. Select Your Function.
3. Choose MyLambdaFunction.

b. Enable Logging

1. Scroll down to the Monitoring section.
2. Click on View logs in CloudWatch.

3. Ensure that the function has the necessary permissions to write logs to CloudWatch (this should be part of the Lambda execution role).

3. Create CloudWatch Alarms for RDS Performance Metrics

Open the CloudWatch Console.

Create Alarm for RDS CPU Utilization

1. In the left-hand navigation pane, click on Alarms.
2. Click Create Alarm.
3. Click Select metric.
4. In the search bar, type RDS and select Per-Instance Metrics.
5. Choose the RDS instance you want to monitor.
6. Select the metric CPUUtilization.
7. Click Select metric.

?

8. Configure the alarm details: Name: RDSCPUUtilizationAlarm Condition: Greater than or equal to 80% Period: 5 minutes

9. Click Next.

10. Configure actions (e.g., send a notification to an SNS topic).

11. Click Next and review the settings.

12. Click Create Alarm.

Create Alarm for RDS Free Storage Space

1. Repeat the steps to create a new alarm.
2. Select the metric FreeStorageSpace.

?3. Configure the alarm details:

Name: RDSFreeStorageSpaceAlarm

Condition: Less than or equal to 1000000000 (adjust as needed)

Period: 5 minutes

4. Click Next and configure actions.

5. Click Next and review the settings.

6. Click Create Alarm.

Conclusion

In this project, I implemented a CI/CD pipeline for deploying a scalable web application on AWS using various AWS services and Infrastructure as Code (IaC) tools. By integrating AWS Amplify for the frontend, AWS Lambda for the backend, and Amazon RDS for the database, I created a highly available and scalable architecture.

Using Terraform, I provisioned the necessary AWS infrastructure, including VPC, subnets, security groups, IAM roles, and policies, ensuring a secure and efficient environment for the application. The CI/CD pipeline, set up with AWS CodePipeline and CodeBuild, automates the deployment process, ensuring that any changes pushed to the GitHub repository are seamlessly built and deployed.

Additionally, AWS CloudWatch provides comprehensive monitoring and logging capabilities, enabling me to track application performance, detect issues, and set up alerts for critical metrics. This ensures that the application remains reliable and performs optimally.