Serverless Microservice CRUD & Load test hands-on

LevelUp! Lab for Serverless

Github: https://github.com/ragerumal/serverless-microservice

Project Summary: Building a Serverless API with AWS

This project provides a comprehensive guide to creating a serverless API using Amazon Web Services (AWS). The high-level design involves utilizing Amazon API Gateway, AWS Lambda functions, and DynamoDB for data storage.

1. High-Level Design:Amazon API Gateway: A collection of resources and methods.Resource: DynamoDBManager, supporting CRUD operations for DynamoDB and additional testing operations.Method: POST, invoking a Lambda function.Lambda Function: Handles API requests and interacts with DynamoDB based on provided payloads.
2. Setup:Creation of Lambda IAM Role: Custom role granting necessary permissions for DynamoDB and CloudWatch Logs.Creation of Lambda Function: Configuring a Lambda function with Python code to handle API requests and interact with DynamoDB.
3. Testing:Verifying the Lambda function by testing with sample payloads, ensuring correct functionality.Load/Performance testing using Postman for scalability evaluation post solution implementation.
4. Deployment:Creation of DynamoDB Table: Setting up the DynamoDB table for data storage.Creation of API: Configuring API Gateway, creating resources, and integrating with the Lambda function.Deployment of the API to a production stage.
5. Running the Solution:Execution of API endpoints for CRUD operations on DynamoDB items. Verification of inserted items in DynamoDB table.
6. Infrastructure deployment using IAC: Introduction to AWS Serverless Application Model (AWS SAM) for improved serverless application development experience. Example usage and configuration of SAM CLI commands for deploying serverless applications.
7. Cleanup:Proper cleanup of resources created during the project, including DynamoDB tables, Lambda functions, and APIs.

This project serves as a comprehensive tutorial for building and deploying serverless APIs on AWS, utilizing key services such as API Gateway, Lambda, and DynamoDB, while also introducing concepts like AWS SAM for further exploration.

Lab Overview And High Level Design

Let's start with the High Level Design. An Amazon API Gateway is a collection of resources and methods. For this tutorial, you create one resource (DynamoDBManager) and define one method (POST) on it. The method is backed by a Lambda function (LambdaFunctionOverHttps). That is, when you call the API through an HTTPS endpoint, Amazon API Gateway invokes the Lambda function.

The POST method on the DynamoDBManager resource supports the following DynamoDB operations:

• Create, update, and delete an item.
• Read an item.
• Scan an item.
• Other operations (echo, ping), not related to DynamoDB, that you can use for testing.

The request payload you send in the POST request identifies the DynamoDB operation and provides necessary data. For example:

The following is a sample request payload for a DynamoDB create item operation:

{
    "operation": "create",
    "tableName": "lambda-apigateway",
    "payload": {
        "Item": {
            "id": "1",
            "name": "Bob"
        }
    }
}        

The following is a sample request payload for a DynamoDB read item operation:

{
    "operation": "read",
    "tableName": "lambda-apigateway",
    "payload": {
        "Key": {
            "id": "1"
        }
    }
}        

Setup

Create Lambda IAM Role

Create the execution role that gives your function permission to access AWS resources.

To create an execution role

• Open the roles page in the IAM console.
• Choose Create role.
• Create a role with the following properties.Trusted entity – Lambda.Role name – lambda-apigateway-role.Permissions – Custom policy with permission to DynamoDB and CloudWatch Logs. This custom policy has the permissions that the function needs to write data to DynamoDB and upload logs.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"Stmt1428341300017",
         "Action":[
            "dynamodb:DeleteItem",
            "dynamodb:GetItem",
            "dynamodb:PutItem",
            "dynamodb:Query",
            "dynamodb:Scan",
            "dynamodb:UpdateItem"
         ],
         "Effect":"Allow",
         "Resource":"*"
      },
      {
         "Sid":"",
         "Resource":"*",
         "Action":[
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
         ],
         "Effect":"Allow"
      }
   ]
}        

Create Lambda Function

To create the function

• Click "Create function" in AWS Lambda Console
• Select "Author from scratch". Use name LambdaFunctionOverHttps , select Python 3.12 as Runtime. Under Permissions, select "Use an existing role", and select lambda-apigateway-role that we created, from the drop down
• Click "Create function"

• Replace the boilerplate coding with the following code snippet and click "Save"

Example Python Code

from __future__ import print_function

import boto3
import json

print('Loading function')


def lambda_handler(event, context):
    '''Provide an event that contains the following keys:

      - operation: one of the operations in the operations dict below
      - tableName: required for operations that interact with DynamoDB
      - payload: a parameter to pass to the operation being performed
    '''
    #print("Received event: " + json.dumps(event, indent=2))

    operation = event['operation']

    if 'tableName' in event:
        dynamo = boto3.resource('dynamodb').Table(event['tableName'])

    operations = {
        'create': lambda x: dynamo.put_item(**x),
        'read': lambda x: dynamo.get_item(**x),
        'update': lambda x: dynamo.update_item(**x),
        'delete': lambda x: dynamo.delete_item(**x),
        'list': lambda x: dynamo.scan(**x),
        'echo': lambda x: x,
        'ping': lambda x: 'pong'
    }

    if operation in operations:
        return operations[operation](event.get('payload'))
    else:
        raise ValueError('Unrecognized operation "{}"'.format(operation))        

Test Lambda Function

Let's test our newly created function. We haven't created DynamoDB and the API yet, so we'll do a sample echo operation. The function should output whatever input we pass.

• Click the arrow on "Select a test event" and click "Configure test events"

• Paste the following JSON into the event. The field "operation" dictates what the lambda function will perform. In this case, it'd simply return the payload from input event as output. Click "Create" to save

{
    "operation": "echo",
    "payload": {
        "somekey1": "somevalue1",
        "somekey2": "somevalue2"
    }
}        

• Click "Test", and it will execute the test event. You should see the output in the console

We're all set to create DynamoDB table and an API using our lambda as backend!

Create DynamoDB Table

Create the DynamoDB table that the Lambda function uses.

To create a DynamoDB table

• Open the DynamoDB console.
• Choose Create table.
• Create a table with the following settings.Table name – lambda-apigatewayPrimary key – id (string)
• Choose Create.

Create API

To create the API

• Go to API Gateway console
• Click Create API

• Scroll down and select "Build" for REST API

• Give the API name as "DynamoDBOperations", keep everything as is, click "Create API"

• Each API is collection of resources and methods that are integrated with backend HTTP endpoints, Lambda functions, or other AWS services. Typically, API resources are organized in a resource tree according to the application logic. At this time you only have the root resource, but let's add a resource next
• Click "Actions", then click "Create Resource

• Input "DynamoDBManager" in the Resource Name, Resource Path will get populated. Click "Create Resource"

• Let's create a POST Method for our API. With the "/dynamodbmanager" resource selected, Click "Actions" again and click "Create Method".

• Select "POST" from drop down , then click checkmark

• The integration will come up automatically with "Lambda Function" option selected. Select "LambdaFunctionOverHttps" function that we created earlier. As you start typing the name, your function name will show up.Select and click "Save". A popup window will come up to add resource policy to the lambda to be invoked by this API. Click "Ok"

• Our API-Lambda integration is done!

Deploy the API

In this step, you deploy the API that you created to a stage called prod.

• Click "Actions", select "Deploy API"

• Now it is going to ask you about a stage. Select "[New Stage]" for "Deployment stage". Give "Prod" as "Stage name". Click "Deploy"

• We're all set to run our solution! To invoke our API endpoint, we need the endpoint url. In the "Stages" screen, expand the stage "Prod", select "POST" method, and copy the "Invoke URL" from screen

Running our solution

• The Lambda function supports using the create operation to create an item in your DynamoDB table. To request this operation, use the following JSON:

{
    "operation": "create",
    "tableName": "lambda-apigateway",
    "payload": {
        "Item": {
            "id": "1234ABCD",
            "number": 5
        }
    }
}        

• 2. To execute our API from local machine, we are going to use Postman and Curl command. You can choose either method based on your convenience and familiarity.

• To run this from Postman, select "POST" , paste the API invoke url. Then under "Body" select "raw" and paste the above JSON. Click "Send". API should execute and return "HTTPStatusCode" 200.To run this from terminal using Curl, run the below

$ curl -X POST -d "{\"operation\":\"create\",\"tableName\":\"lambda-apigateway\",\"payload\":{\"Item\":{\"id\":\"1\",\"name\":\"Bob\"}}}" https://$API.execute-api.$REGION.amazonaws.com/prod/DynamoDBManager        

• To validate that the item is indeed inserted into DynamoDB table, go to Dynamo console, select "lambda-apigateway" table, select "Items" tab, and the newly inserted item should be displayed.

• To get all the inserted items from the table, we can use the "list" operation of Lambda using the same API. Pass the following JSON to the API, and it will return all the items from the Dynamo table

{
    "operation": "list",
    "tableName": "lambda-apigateway",
    "payload": {
    }
}        

We have successfully created a serverless API using API Gateway, Lambda, and DynamoDB!

Performance test using Postman

• Install native desktop version postman tool and sign in to an account.
• Create new collection and post method like below

• List operation below

• Run collection in performance test mode

• Sample report for the performance test done

Infrastructure deployment:

The AWS Serverless Application Model (AWS SAM) is a toolkit that improves the developer experience of building and running serverless applications on AWS.

AWS SAM template specification – An open-source framework that you can use to define your serverless application infrastructure on AWS.

1. Sample SAM template as below

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'

Description: SAM template for API Gateway, Lambda, and DynamoDB CRUD project architecture

Resources:
  DynamoDBTable:
    Type: 'AWS::DynamoDB::Table'
    Properties:
      TableName: lambda-apigateway
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5

  LambdaFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: python3.8
      Policies:
        - DynamoDBCrudPolicy: {}
      Environment:
        Variables:
          TABLE_NAME: !Ref DynamoDBTable
      Events:
        ApiGatewayEvent:
          Type: Api
          Properties:
            Path: /dynamodbmanager
            Method: post

Outputs:
  ApiUrl:
    Description: URL of the API Gateway endpoint
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/dynamodbmanager'
  DynamoDBTableName:
    Description: Name of the DynamoDB table
    Value: !Ref DynamoDBTable
        

1. sam deploy --guided --template template.yaml is the command you enter at the command line.
2. sam deploy --guided --template should be provided as is.
3. template.yaml can be replaced with your specific file name.
4. The output starts at Configuring SAM deploy.
5. In the output, ENTER and y indicate replaceable values that you provide.

Cleanup

Let's clean up the resources we have created for this lab.

Cleaning up DynamoDB

• To delete the table, from DynamoDB console, select the table "lambda-apigateway", and click "Delete table"

• To delete the Lambda, from the Lambda console, select lambda "LambdaFunctionOverHttps", click "Actions", then click Delete

• To delete the API we created, in API gateway console, under APIs, select "DynamoDBOperations" API, click "Actions", then "Delete"