How to use S3 securely

Amazon's S3 is one of the most widely used cloud object storage and retrieval services on the Internet. It underpins a large proportion of many Web apps and services, and is also used by enterprise customers due to its scalability and availability. There is however a problem with it, or rather with how it is sometimes used.

S3 can be used in a way which is very insecure, and in ways which have different degrees of security. In this short post I would like to discuss and contrast two of the ways you can use S3: the insecure way and one of the most secure ways.

First, the insecure way. The most insecure way to use S3 is to have an easily guessable S3 bucket name, which is part of a well-known URL, and have unauthenticated public access policy. That would mean that anyone on the Internet with the S3 URL for a particular object (file) can issue an HTTP GET request and download the object without any further ado.

The only time when such policy may be appropriate is when you have a bucket which houses exclusively publicly available information which you don't mind anyone downloading as often as they want.

If your bucket contains anything other than public data, such an access policy is a disaster waiting to happen, since S3 URLs can and are discovered, and once an attacker has the URL to an S3 object without any access control they have your data.

So, if you have any data which is not public, you need to apply an appropriate access control policy to your S3 bucket which should specify the requirements for both upload and download operations. There are different ways to do it, but perhaps the best one for Internet-facing Web apps is the pre-signed URL mechanism.

Pre-signed URLs in Amazon S3 are a powerful tool for providing temporary access to private objects stored in S3 buckets: when you generate a pre-signed URL, you essentially create a URL that includes your access key, an expiration time, and a cryptographic signature. This signature is calculated using your AWS secret access key and it authorises the URL for a specific HTTP operation (like GET, POST or PUT) on a specific S3 object.

The key aspect of this mechanism is that once generated, the URL can be used by anyone who has it to perform the operation it was created for (such as the end user), without needing additional AWS credentials.

The URL remains valid until the specified expiration time is reached (which could be as short or as long as required), after which it becomes invalid and the object cannot be accessed using that URL.

This makes pre-signed URLs particularly useful for controlled sharing of files, enabling scenarios such as temporary access for downloading a file or uploading to a specific location in an S3 bucket, without exposing broader permissions to the user.

It is important to add that whatever access mode you use for S3, you should enable access logging, to know what is downloaded or uploaded to your buckets.

By default Amazon does not generate logs, which is a nightmare if you have a security incident and need to investigate what has been accessed and when.

There is no extra charge for generating logs but there is a charge for storing them, so you need to make sure you implement an appropriate log retention policy, such as 30, 60 or 90 days.

For applications with higher security requirements, AWS CloudTrail can also be used. When used in conjunction with S3, CloudTrail enhances security and compliance by logging API calls and related events that occur within S3, providing more detailed information about the requests made to an S3 bucket.

Finally, here's an interesting post from Censys which illustrates the extent of the problem by discussing how insecure S3 buckets can and are being found: https://censys.com/using-censys-to-find-misconfigured-s3/

I have been engineering and testing AWS security for over 11 years. If you have an application hosted on AWS and need to obtain independent security assurance for it, book a meeting to discuss how I can help.