Automating Security Incident Responses in AWS

I am an automation junkie on both the Development side and Operations side of DevOps and everything in-between. I even automate Data Pipelines and MLOps. So, it just come natural to me. Security seems to be a place where lights-out automation seems to be lagging. There's ton's of tools, scanners, and monitors, but they seems to focus on the front end up to the point of detection and then a human is required. One of the things I love about AWS is everything is code and programmable and AWS provides more instrumentation and telemetry and I've seen in most on-premises security architectures.

In this article we will look at a specific security incident type and how we can completely automate a playbook for addressing the threat, performing root cause analysis of "how" and some additional environmental hardening that can be done based on the RCA.

The Incident

In this case we have an EC2 instance that is experiencing a brute force attack and IAM credentials that have been compromised. The AWS services that we will employ to automate the response include:

• AWS GuardDuty
• Amazon EventBridge
• AWS StepFunctions
• AWS IAM
• AWS System Manager - Session Manager
• AWS CloudWatch
• AWS Security Hub
• Amazon Inspector
• Lambda
• NACLs
• Security Groups

The Goals

What are expectations for an automated response? 1) We want to be able to detect the threat and automatically revoke, rotate, and verify the access keys. 2) We need to check to see if the attack was successful or not and based on what we learn, 3) take long-term corrective action to ensure it can't happen again and also save all of the relevant data for additional forensics.

A Solution

Here is one possible way to do this:

• We need AWS GuardDuty set up to monitor and alert in the event of a brute force attack on our EC2 instances. AWS GuardDuty can identify the compromised IAM role
• We will use Amazon EventBridge rules to manage our actions when it received an event from AWS GuardDuty. Each rule will trigger an AWS Step Function. The AWS Step Functions are our automated playbook that will orchestrate a series of steps for how we wish to handle this particular incident.
• First we want to immediately secure the EC2 Instance. AWS GuardDuty's alert will provide the EC2 InstanceID being attacked and the IAM role that is compromised. We will call a Lambda function (Step 1) that will use IAM and AWS KMS to create new access keys for the EC2 instance, revoke the old key and restart the EC2 Instance in order to rotate the access keys.
• We will invoke another Lambda function (Step 2) to use AWS Systems Manager's Session Manager to verify that the access keys were properly rotated on the restarted EC2 instance.
• Second we want to do some investigating. Now that the impacted EC2 Instance is back in a secure state and running, we will invoke a Lambda function (Step 3) that will check AWS CloudWatch to see if the attack was successfully by looking for successful logins.
• Additionally we will have a Step Function (Step 4) use Amazon Inspector to perform a scan of the EC2 Instance. The results will be available in AWS Security Hub.
• Last we need to act on what we've learned to prevent future incidents. A Lambda function (Step 5) will lookup the EC2 Instance ID in Security Hub for communications from IP addressed on disallowed list (i.e. Brute force attack). It will also use Security Hub to view the findings of the Amazon Inspector scan and based on all of the information available it can update network ACLs or Security Groups accordingly.
• One of my best practices it to incorporate some Slack integration so that the automation can report to a channel what it's detected, what it found, and event to get approval to proceed with each step.

Regardless of how you do it, I hope this shows that AWS provides a lot of tools and functionality to really up your security game in the cloud. I would write up a playbook for each type of vulnerability and incident you need immediate responses to and then use the rich set of capabilities in AWS to automate it.

Thanks for reading.