AWS Security Groups - The Good, the bad and the Ugly

AWS security groups have been around for ages. They are one of the fundamental controls in a VPC (virtual private cloud). You can attach them to so many products (like ec2, RDS etc). I though it was worth re-visiting them based on recent experience...

What is a security group (and the good points)?

Security groups are pretty much firewalls. They are a great way of protecting resources. They can be applied to individual resources so rather than having one firewall protecting your ingress, you can have a separate firewall protecting each individual resource. This is a big improvement over traditional firewalls. Not only does it protect your resources from internal traffic, it also greatly simplifies the management of your firewall rules.

• Security groups can reference other security groups. This is a really powerful concept. In a rapidly changing and scalable cloud infrastructure, it is often not possible to use IP address ranges to identify resources. For example, if you create one security group for your web servers and another one from your database, it is easy to say everything in the web server group can access the database. They can also reference IP ranges for traffic originating from outside your AWS account.
• Security groups have ingress (inbound) and egress (outbound) rules. There are 2 separate lists of rules.
• Security groups are stateful. That means they allow reply traffic automatically. There is no need to be adding separate rules for unprivileged ports.
• Security groups are all allow rules. By default all traffic is denied. You just add the rules for allowed traffic. Most of the time this is not a problem and it makes the structure of security groups very easy. Unlike many firewalls, the order of rules does not matter as they are all allow. Readability of security group rules is greatly beneficial for avoiding inadvertent security group holes!
• Security groups can even be used across accounts. If you have VPC peering between 2 VPC in different accounts, you can reference security groups in the other account. There are some slight limits to the functionality (around reference checking), but it pretty much works the same.

As well as security groups, there are Network Access Control Lists (nACL). These can be used in addition to security groups. NALC are a perimeter control, they are not stateful and have both allow and deny lists. It this article I am focussing on security groups but it is worth mentioning nACL also. In general I would recommend using security groups as the 'primary' control and then use nACL as either a 'belt and braces' approach or if you need to add explicit deny rules.

The bad points

There are not many bad points to security groups:

There are some account/vpc limits. It is normally rare to reach them. They can be viewed here https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html In most cases if you are hitting the security group limits you may want to split up your services between separate AWS accounts anyway. This can be good from a security point of view and helps reduce blast radius in case of an issue.

As mentioned in the good points security groups are default deny and only has allow rules. In 99% of cases this is absolutely fine. If you do need an explicit deny rule, then you have to do that using VPC network access control lists (nACL). That sometimes takes a bit of planning when you have a very dynamic network. Generally though other tools like AWS shield (and associated products) should be used to block attacks. It is very rare to use explicit deny rules.

The ugly

There are a couple of security group features that conspire to annoy you with security groups.

• You must have at least one security group assigned to most network interfaces that use security groups
• You need to remove references to security groups before you can delete them.

If you have 2 security groups that reference each other it can make deleting them quite interesting. Most of the problems come when trying to use infrastructure as code (which you really should be doing). You can create dependancies that can't be resolved and cause changes to fail. One saving grace is that a lot of this can be used by using a create before destroy lifecycle. Rather than altering a security group a new one is created. It is then assigned to the relevant resource. Then the old security group can be removed. This can generally be done with zero outage but you do have to check if you can stand a service being unavailable even momentarily.

There are 2 other annoyances with security groups. One of which is minor, the other is a major pain:

• The first is that some fields on a security group can't be updated. The security group needs to be recreated. The name is one of those fields. It is a minor annoyance but worth being aware of. Sometimes you just want to improve the labelling and it is a pain.
• The other issue is the compatibility with Transit Gateways when used across accounts. At present you can't create cross account references when using transit gateways like you can with VPC peering. This is a serious drawback. From what I understand this does require a major change behind the scenes and has been on the AWS wish list for a long time.

Security group changes are a common reason why IaC updates can fail. Minor changes like a simple name change can fail because of references to the security group. Create before deleting fixes may items, but the one time when you can have major issues is when you have multiple IaC stacks for different elements of your environments and they reference each other. There are a few ways to resolve these issues but it does feel more painful than it should be.

Conclusion

Security groups are one of the most powerful networking concepts in the AWS world. They can be very useful for creating security in depth rather than just perimeter security. There its a real advantage in simplifying security. Many security issues are created by complexity. There are a

• Think about the name and function of your security groups before you create them
• Plan boundaries of IaC stacks carefully. Ideally have a product contained in one stack and only expose a single entry point if possible
• Always add descriptions to security groups and rules. As much as possible try and communicate the intent. When you come back to look at a rule in the future you may not recognise an IP address that was added so rule descriptions are essential
• Create separate rules for different services. It can too easy to lump too many services into a security group. This can lead to much more permissive security
• Think about how you are going to provide admin or emergency access if you need to.