S3 Cleanup in AWS to Unlock $$

Incomplete (mulit-part-uploads) MPU objects can occur if your automated job that uploads artifacts to a S3 bucket fails for an unexpected reason (ex. network error).

That means that part of the data which has already been uploaded to S3, and will still be stored there, even if that specific upload session does not resume adequately.

Amazon S3 Storage Lens offers an advanced feature for understanding company-wide object-storage behaviors and tendencies. This tool not only provides significant data metrics but also offers informed suggestions which can be utilized to streamline storage expenses and ensure data security measures are in place.

With Amazon S3 Storage Lens data metrics, you can gain overarching insights such as quantifying the storage used throughout your company or pinpointing the most rapidly expanding storage areas and sections. Additionally, these metrics can assist in uncovering opportunities to reduce costs, promote robust data safety, and enhance the efficiency of application tasks.

For instance, it's possible to detect storage areas lacking automated rules to terminate unfinished data uploads older than a week (MPUs).

Furthermore, you can highlight storage segments not adhering to optimal data safety protocols like using data duplication or version control features.

Amazon S3 Storage Lens consolidates your metrics and showcases them under the Account Overview area of the Cloud Storage main console page. This analytics tool also introduces an intuitive interface, enabling you to graphically represent data patterns, pinpoint anomalies, and gain suggestions for cost reductions and data safeguarding. The interface comes with detailed analysis capabilities to view data at various levels, from company-wide down to specific storage segments or units. Additionally, there's an option to automate a daily data report in formats like CSV or Parquet, delivered straight to a storage bucket.

Look for MPUs that are taking up space, and also result in added charges monthly that you can mitigate by doing some spring cleaning.

Following that, you can write a policy that does some basic housekeeping to avoid MPUs from clogging up your S3 storage going forward.

Creating this MPU retention policy will help with your CI operations.

Go to AWS S3 Console (check Management tab on specific S3 bucket page), bu utilizing AWS Command Line Interface (AWS CLI).

Set MPU retentions, by doing the following:

1. Ensure that you have AWS CLI installed + configured.
2. Now prepare the following mpu-retention.json file:

{
    "Rules": [
        {
            "ID": "MPU Retention",
            "Status": "Enabled",
            "Filter": {
                "Prefix": ""
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}        

It describes the policy to abort all incomplete upload sessions that were not active within the past week.

3. Run the following command to apply the MPU retention policy to your-bucket:

aws s3api put-bucket-lifecycle-configuration --bucket your-bucket --lifecycle-configuration file://mpu-retention.json        

With the MPU retention rule in place, the MPUs won't build further. Note that AWS lifecycle policies are applied once per day, so allow 24 hours for the new policy to take.

For more ways to cleanup S3, see the AWS article dedicated to this topic here.

To learn how?CDW?can help you save even more in AWS, GCP or Azure. We recommend talking to us about our?FinOps Accelerator: