Exporting AWS RDS Snapshots to S3 Using Lambda Functions

In the world of cloud computing, Amazon Web Services (AWS) offers a robust ecosystem of services that can be combined to create powerful solutions. Three core components of this ecosystem — Amazon RDS, Amazon S3, and AWS Lambda — can be used in tandem to automate the process of exporting RDS snapshots to an S3 bucket. In this article, we’ll explore how to achieve this using Lambda functions.

Introduction to AWS Services

Before diving into the specifics of exporting RDS snapshots to an S3 bucket via Lambda, let’s briefly introduce the key AWS services involved:

Amazon RDS (Relational Database Service)

Amazon RDS is a managed relational database service that simplifies database setup, operation, and scaling. It supports various database engines such as MySQL, PostgreSQL, SQL Server, and more. RDS allows you to create database instances and take snapshots of your data, which are point-in-time backups.

Amazon S3 (Simple Storage Service)

Amazon S3 is a scalable and highly available object storage service. It provides secure, durable, and cost-effective storage for various data types, including snapshots. S3 is organized into “buckets,” which are containers for storing objects, and each object is identified by a unique key.

AWS Lambda

AWS Lambda is a serverless computing service that enables you to run code in response to events. You can use Lambda to execute functions that interact with other AWS services, like RDS and S3, without the need to manage servers.

Exporting RDS Snapshots to S3

I will address three scenarios for exporting RDS snapshots to an S3 bucket using Lambda functions.

1. Exporting a Snapshot of an RDS DB Instance

This scenario involves creating a new snapshot of an RDS DB instance and then exporting it to a specific S3 bucket using a Lambda function.

import boto3
from datetime import datetime
def lambda_handler(event, context):
    # Enter the RDS details
    client = boto3.client('rds')
    db_identifier = 'your-database-instance-name'
    snapshot_identifier = 'snapshot-' + datetime.today().strftime('%Y-%m-%d-%H-%M-%S')
    s3_bucket_name = 'your-s3-bucket-name'
    s3_bucket_folder = 'snapshots/'
    iam_role_arn = 'xxx' # Replace with your IAM Role ARN
    kms_key_id = 'xxx'  Replace with your KMS key
    # Create snapshot
    client.create_db_snapshot(DBSnapshotIdentifier=snapshot_identifier, DBInstanceIdentifier=db_identifier)
    # Create RDS DB snapshot waiter
    waiters = client.get_waiter('db_snapshot_completed')
    waiters.wait(DBSnapshotIdentifier=snapshot_identifier)
    # Export snapshot to S3
    export_task_identifier = 'export-' + datetime.today().strftime('%Y-%m-%d-%H-%M-%S')
    s3_export_task = client.start_export_task(
        ExportTaskIdentifier=export_task_identifier,
        SourceArn='arn:aws:rds:region:123456789:snapshot:'+snapshot_identifier, # Replace the region and the account id with your own
        S3BucketName=s3_bucket_name,
        IamRoleArn=iam_role_arn,
        KmsKeyId=kms_key_id,
        S3Prefix=s3_bucket_folder
    )
    return {
        'statusCode': 200,
        'body': 'Snapshot exported successfully!'        

2. Exporting an Existing RDS Instance Automatic Snapshot

In this case, we will export an existing automatic snapshot of an RDS instance to your designated S3 bucket using Lambda.

import boto3
from datetime import datetime

def lambda_handler(event, context):
    # Enter the RDS details
    client = boto3.client('rds')
    db_instance_identifier = 'your-db-instance-name'
    snapshots = client.describe_db_snapshots(DBInstanceIdentifier=db_instance_identifier, SnapshotType='automated')
    
    if not snapshots['DBSnapshots']:
        return {
            'statusCode': 404,
            'body': 'No snapshots found for instance identifier {}'.format(db_instance_identifier)
        }
    
    latest_snapshot = sorted(snapshots['DBSnapshots'], key=lambda x: x['SnapshotCreateTime'], reverse=True)[0]
    snapshot_identifier = latest_snapshot['DBSnapshotIdentifier']
    s3_bucket_name = 'your-s3-bucket-name'
    s3_bucket_folder = 'snapshots/'
    iam_role_arn = 'xxx' # Replace with your IAM Role ARN
    kms_key_id = 'xxx' # Replace with your KMS key
    
    # Export snapshot to S3
    export_task_identifier = 'export-' + datetime.today().strftime('%Y-%m-%d-%H-%M-%S')
    s3_export_task = client.start_export_task(
        ExportTaskIdentifier=export_task_identifier,
        SourceArn='arn:aws:rds:region:123456789:snapshot:'+snapshot_identifier, # Replace the region and the account id with your own
        S3BucketName=s3_bucket_name,
        IamRoleArn=iam_role_arn,
        KmsKeyId=kms_key_id,
        S3Prefix=s3_bucket_folder
    )
    return {
        'statusCode': 200,
        'body': 'Snapshot exported successfully!'
    }        

3. Exporting an Existing RDS Cluster’s Automatic Snapshot

When dealing with RDS clusters, you can export the latest automatic snapshot of a cluster to an S3 bucket via Lambda.

import boto3
from datetime import datetime

def lambda_handler(event, context):
    # Enter the RDS details
    client = boto3.client('rds')
    db_cluster_identifier = 'your-database-cluster'
    snapshots = client.describe_db_cluster_snapshots(DBClusterIdentifier=db_cluster_identifier, SnapshotType='automated')
    latest_snapshot = sorted(snapshots['DBClusterSnapshots'], key=lambda x: x['SnapshotCreateTime'], reverse=True)[0]
    snapshot_identifier = latest_snapshot['DBClusterSnapshotIdentifier']
    s3_bucket_name = 'your-s3-bucket-name'
    s3_bucket_folder = 'snapshots/'
    iam_role_arn = 'xxx' # Replace with your IAM Role ARN
    kms_key_id = 'xxx' Replace with your KMS key
    
    # Export snapshot to S3
    export_task_identifier = 'export-' + datetime.today().strftime('%Y-%m-%d-%H-%M-%S')
    s3_export_task = client.start_export_task(
        ExportTaskIdentifier=export_task_identifier,
        SourceArn='arn:aws:rds:region:123456789:cluster-snapshot:'+snapshot_identifier, # Replace the region and the account id with your own
        S3BucketName=s3_bucket_name,
        IamRoleArn=iam_role_arn,
        KmsKeyId=kms_key_id,
        S3Prefix=s3_bucket_folder
    )
    return {
        'statusCode': 200,
        'body': 'Snapshot exported successfully!'
    }        

Conditions for Storing Snapshots in S3

To ensure that your RDS snapshots are effectively stored in your S3 bucket, consider the following best practices:

1. Access Control: Implement AWS Identity and Access Management (IAM) policies to control who can access and modify the S3 bucket.
2. Encryption: Use AWS Key Management Service (KMS) to encrypt your snapshots stored in S3 for data security.
3. Versioning: Enable versioning on your S3 bucket to keep multiple versions of snapshots for backup and recovery purposes.
4. Lifecycle Policies: Define lifecycle policies to automatically transition older snapshots to cheaper storage classes, like Amazon S3 Glacier, after a certain period.
5. Monitoring and Alerts: Set up CloudWatch alarms to monitor the health and performance of your Lambda functions, RDS instances, and S3 buckets.

By adhering to these best practices, you can ensure the reliability, security, and cost-effectiveness of your RDS snapshot exports to S3.

Conclusion

Automating the process of exporting RDS snapshots to an S3 bucket using AWS Lambda is a valuable addition to your cloud infrastructure toolkit. It enables efficient backup and archiving of your database snapshots, enhancing data durability and availability.