AWS Infrastructure Compliance Audit System with AWS Config and Lambda

In this project, you will use AWS Config to monitor the configurations of various AWS resources (like EC2 instances, S3 buckets, IAM policies) for compliance with predefined rules. An AWS Lambda function will be used to send alerts or take action when resources are found to be non-compliant.

Step-by-Step Details:

Step 1: AWS Config Setup

Open AWS Config:

Navigate to the AWS Config console. You can find it by typing “Config” in the AWS services search bar.

Set Up AWS Config (if it’s your first time):

If you're using AWS Config for the first time, you'll be greeted with a setup page. Click on “Get started” to begin the setup process.

Choose Resources to Record:

You can select specific types of resources (like EC2 instances, S3 buckets, IAM policies) or choose to record all resources in your AWS account. Decide based on what is relevant for your compliance needs.

Choose an IAM Role:

AWS Config requires an IAM role to record configuration changes. You can allow AWS Config to create a role on your behalf or select an existing role that has the necessary permissions.

Configure AWS Config to Monitor Desired Resources

Set Up the S3 Bucket:

AWS Config needs an S3 bucket to store configuration history and snapshots. You can create a new S3 bucket or choose an existing one. Ensure that the bucket is properly configured for security and access control.

Select Resources:

After the basic setup, you’ll need to select the specific resources you want to monitor.

AWS Config allows you to choose from a variety of AWS resources like EC2 instances, VPCs, S3 buckets, etc.

Add Rules

Choose from the list of available managed rules. For custom rules, you'll need to define the rule in AWS Lambda and then add it here.

Review and Confirm:

Once you have selected your resources, rules and set up the recording options, review your choices. Confirm and enable AWS Config.

Once you have completed these steps, AWS Config will start recording the configurations of the resources you selected and track any changes made to them. This is the foundation for the compliance monitoring system you're building.

Step 2: AWS Lambda Function for Alerts

This stage requires the following:

Amazon SNS Topic: You should have an Amazon SNS topic created to which the alert messages will be published. You can create one in the Amazon SNS console.

Amazon SES: If you opt to send an email, make sure you have verified an email address in Amazon Simple Email Service (SES) to send emails from.

IAM Role: Your Lambda function will need an IAM role with permissions to access AWS Config, publish messages to Amazon SNS, send emails via SES, and write logs to CloudWatch.

Python Lambda Function Code:

Below is the Python code that I will be using for my AWS Lambda function. It will send a notification to an SNS topic when a non-compliant resource is detected:

import json
import boto3

# Initialize boto3 client for SNS
sns_client = boto3.client('sns')

def lambda_handler(event, context):
    # Parse the JSON message
    message = json.loads(event['Records'][0]['Sns']['Message'])
    
    # Print the message to see what data you received from AWS Config
    print("Message: ", json.dumps(message, indent=2))
    
    # Extract information from the message
    compliance_type = message['newEvaluationResult']['complianceType']
    resource_type = message['resourceType']
    resource_id = message['resourceId']
    
    # Check if the resource is non-compliant
    if compliance_type == "NON_COMPLIANT":
        # Construct an alert message
        alert_message = f"AWS Config detected a non-compliant resource.\nResource Type: {resource_type}\nResource ID: {resource_id}"
        
        # Log the alert message
        print("Alert: ", alert_message)
        # (Here, you could also send another notification, log to CloudWatch, or take remediation actions)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Lambda function executed successfully!')
    }        

Deployment Steps:

1.????? Create a new Lambda function:

Go to the AWS Lambda console.

Choose to create a new function.

Select "Author from scratch".

Enter the function name "ConfigComplianceAlertFunction".

Choose the Python 3.x runtime.

Select or create an IAM role with the appropriate permissions.

2. Function Code:

Copy and paste the Python code provided above into the Lambda code editor. Replace SNS_TOPIC_ARN with the ARN of your SNS topic.

3. Configure Trigger:

Add a trigger to the function. Select SNS as the trigger source. Choose the SNS topic you created for AWS Config alerts.

4.????? Configure AWS Config to Send Alerts to the SNS Topic:

In the AWS Config console, go to the rule you've created. Specify an SNS topic that AWS Config will send a message to when the rule detects a non-compliant resource.

Once this setup is complete, your AWS Config will send all compliance change notifications to the specified SNS topic. This topic, in turn, will trigger the subscribed Lambda function, allowing you to handle the notification according to your function's logic.

5. Test the Function:

You can simulate an AWS Config event in the Lambda console to test your function. Adjust the test event to mimic a non-compliant resource event from AWS Config.

I will be using this JSON in the Lambda console to test the function.

{
  "Records": [
    {
      "Sns": {
        "Message": "{\"newEvaluationResult\":{\"complianceType\":\"NON_COMPLIANT\",\"resourceType\":\"AWS::S3::Bucket\",\"resourceId\":\"example-bucket\"}}"
      }
    }
  ]
}        

6. Deploy the Function:

Once you're satisfied with the function and its tests, deploy the function. It will now automatically run when AWS Config detects non-compliant resources.

This script will receive the event from AWS Config, parse it to determine if the resource is non-compliant, and if so, send an alert message to the specified SNS topic.

Step 3: Permissions and Role

Navigate to the lambda IAM role and attach the necessary permissions for the Lambda function to access AWS Config data and send alerts via your chosen notification method. Add the following permissions: “AWSLambdaBasicExecutionRole,” “AWSConfigRulesExecutionRole” and “AmazonSNSFullAccess.” For more fine-grained control, you can create a custom policy that only allows publishing to a specific SNS topic.

If you are using Amazon SES to send emails, attach AmazonSESFullAccess. Again, a custom policy can be created for tighter security, allowing only the sending of emails.

Step 4: Testing and Validation

Test your setup by creating resources that violate your Config rules and ensure that your Lambda function is triggered.

Based on the AWS Config rules we have listed, here are a couple of options I will be using to modify the setup:

s3-bucket-logging-enabled:

This rule checks if logging is enabled for your S3 buckets.

To test this, navigate to the S3 console, select a bucket that is currently compliant (i.e., it has logging enabled), and then disable the logging feature.

s3-bucket-level-public-access-prohibited:

This rule checks if your S3 buckets are publicly accessible.

Go to the S3 console, select a bucket, and modify its permissions to allow public access. Do this by editing the bucket policy or the ACL (access control list).

Verify that you receive the configured alerts and that the information is accurate.

Troubleshooting AWS Config and SNS Notification Issues

After implementing AWS Config rules to monitor compliance of resources, I anticipated receiving SNS notifications upon non-compliance. However, notifications were not received as expected. The following steps were taken to diagnose and resolve the issue:

1. AWS Config Rule and Role Verification: Confirmed that AWS Config rules were active and evaluations were being triggered. Inspected CloudWatch logs and addressed an error related to exceeding the maximum request rate by reducing the number of rules to focus on critical ones for this test. Despite these adjustments, SNS notifications were still not forthcoming.
2. SNS Topic Configuration Check: Verified the SNS topic was configured correctly to receive AWS Config notifications. Ensured the SNS topic access policy granted AWS Config permission to publish messages. The review confirmed that the access policy was appropriately set up, allowing AWS Config to interact with the SNS topic.
3. Lambda Function Permissions: Checked and confirmed that the Lambda function had the necessary permissions to be triggered by the SNS topic. The execution role and resource-based policy for Lambda (invoking sns.amazonaws.com) were set correctly.
4. CloudWatch Log Review: No SNS-related activity was detected in CloudWatch logs, only logs from manual Lambda function invocations during testing.
5. S3 Bucket Configuration and SNS Subscription: Confirmed that logging and public access block configurations were enabled on the S3 buckets under observation. Ensured that my email was correctly subscribed to SNS notifications.

Despite thorough checks and confirmations, the anticipated SNS notifications were not received. Upon further investigation, I discovered the root cause:

Resolution:

• Identified that the default service-linked role selected during AWS Config setup—"Use an existing AWS Config service-linked role"—lacked the necessary permissions for actions from Amazon S3, Lambda, and SNS.
• Created a new IAM role tailored for AWS Config, attaching the necessary permissions to facilitate proper actions from Amazon S3, Lambda, and SNS services.
• Configured AWS Config to utilize this new role.

Outcome:

After linking the new role with enhanced permissions to AWS Config, notifications began to flow in as expected. This resolution brought the system to operational status, enabling the receipt of SNS notifications upon resource compliance changes, confirming the effectiveness of the troubleshooting process and the implemented solution.

Step 6: Monitoring and Logging

Never set up a system without monitoring and logging

Navigate to 'Logs' in the CloudWatch console.

Find and select the log group associated with your Lambda function. The name typically follows the pattern /aws/lambda/function-name.

Review Execution Logs:

Within the log group, you will see a series of log streams. Click on a log stream to view the logs generated by specific executions of your Lambda function.

Set Up CloudWatch Alarms

Navigate to Alarms:

In the CloudWatch console, go to 'Alarms'. Click on 'Create Alarm'.

Select Metric:

Click on 'Select metric'.

Navigate to 'Lambda', then select 'By Function Name'. Choose the metric dimensions relevant to your function, such as 'Errors', 'Invocations', 'Duration', etc.

Select the metric to alarm on, such as 'Errors', and click 'Select metric'. You can only choose one for each alarm.

Configure Alarm:

Define the alarm condition. For example, you can set the alarm to trigger if 'Errors' is 'Greater/Equal' to 1 for 1 consecutive period(s). Specify the period over which the metric is evaluated (e.g., 1 minute).

Set Notification:

Under 'notification', configure what happens when the alarm state is reached. Usually, this involves sending a notification to an SNS topic.

If you haven't created an SNS topic for alarms, you will need to create one. Go to the SNS console, create a topic, and then subscribe to it using your email or another endpoint.

Name and Create Alarm:

Give your alarm a meaningful name and description. Click on 'Create Alarm'.

Once you have set up CloudWatch Alarms, you will be notified if the specified conditions are met, indicating potential issues with your Lambda function. Regularly reviewing your CloudWatch Logs will also give you insights into the performance and behavior of your Lambda function, allowing you to proactively address any issues that arise.

Project Wrap-Up: AWS Infrastructure Compliance Audit System

As I conclude the implementation of our AWS Infrastructure Compliance Audit System, it's clear that diligent monitoring and governance are not just regulatory requirements—they are integral to our operational integrity in the cloud.

Leveraging AWS Config's managed rules allowed us to quickly establish a baseline for compliance, while custom Lambda functions provided targeted oversight, ensuring efficiency and proactive management of our cloud resources.

Key takeaways from this project include the value of iterative review processes, the importance of precise error handling in automated responses, and the significant cost benefits of efficient Lambda function execution.

This initiative serves as a reminder of the ongoing nature of compliance—it is a continuous commitment. To that end, I'm extending an invitation for collaboration and shared insights. The collective expertise will be invaluable as we refine our compliance strategies and adapt to evolving regulations.

I welcome dialogue and insights on this and future projects. Together, let's set a benchmark for cloud compliance and security excellence.

?