Implementing S3 Conditional Writes

Introduction

Amazon Simple Storage Service (S3) is a highly scalable object storage service that provides secure, durable, and cost-effective storage for virtually any kind of data. It's known for its high availability, global redundancy, and ability to store massive amounts of unstructured data. S3 is commonly used for data lakes, backup and restore, content distribution, and big data analytics. For many services such as AWS Glue, Athena, Sagemaker, EMR (EMRFS), Redshift (Spectrum), Bedrock KBs, Backup services etc the underlying storage used is S3. The design principles of S3 are based on the following

Immutability: Objects in S3 are immutable, meaning once they are written, they cannot be modified. Changes require overwriting the entire object or creating new versions if versioning is enabled.

Unique Keys: Each object in an S3 bucket is identified by a unique key, which acts as the address for accessing the object.

Eventual Consistency: S3 was originally designed with eventual consistency, meaning there might be a bit of delay in object availability or consistency after updates. However, as of 2020, S3 provides strong read-after-write consistency for new object creation and updates.

Versioning: S3 supports versioning, allowing multiple versions of the same object to coexist, which enhances data recovery capabilities.

Event-Driven Patterns: S3 integrates with services such as AWS Lambda to enable event-driven workflows. Events such as object creation, deletion, or updates can trigger Lambda functions, supporting real-time processing.

It should be noted that S3 is an object storage service and was never intended to provide file system type semantics such as distributed locks, strong consistency, in-place updates as is common with POSIX compliant file systems. For truly distributed locking requirements, AWS provides EFS, FSx and you may also uses DynamoDB along side of S3 for implementing distributed locks.

Since its introduction in 2006, S3 has added severla major features notably cross region replication (CRR), S3 transfer acceleration, intelligent tiering, muli region access points, object Lambda and most recently S3 conditional writes. In this blog we will examine what is S3 conditional write, what are its use cases, when the S3 conditional writes are not a good fit to use and finally what makes this an excellent feature addition to S3.

What are S3 Conditional Writes?

S3 conditional writes are introduced to prevent accidental overwrites of an object when multiple write operations are done on the same object concurrently. First and foremost, the very idea of conditional writes can cause confusion on several fronts. Some obvious questions are isn't this an anti pattern of S3 design principles? Didn't AWS already introduce object locks and strong consistency in 2020? Are conditional writes force fitting file system semantics on S3 especially when this is not suitable for distributed locking type of scenarios? Some of these questions are due to misunderstood context of what S3 conditional writes are. Lets delve a bit deeper into this aspect.

At the outset S3 conditional writes are not distributed locks, these are atomic compare-and-swap (CAS) operations and based on optimistic concurrency control. Key distinction between a lock and CAS atomic operation are essential to understand how S3 conditional writes work. A lock from file system perspective holds the resource for a specific duration during which it may be released or can timeout based on the value set and/or has potential to enter deadlock condition. CAS on the other hand is an atomic operation which can succeed or fail, and can be retried several times. Simply put the conditional writes are CAS operations where writes are coordinated without locking.

In summary Compare-And-Swap (CAS) is a common atomic operation in computer science that is used to manage concurrent access to a shared resource without requiring locks. In the context of object storage, CAS operations ensure that updates are only applied if certain conditions are met, which can help prevent conflicting changes.

To summarize the CAS functionality of S3 can be understood the following way.

Compare - The ETag value serves as the compare part. S3 checks if the current object's ETag matches the one provided in the IfMatch condition

Swap - If the ETag matches (comparison succeeds), the new object content is written and new ETag is associated. If the ETag doesn't match, the operation fails with PreconditionFailed.

S3 Conditional Writes Use Cases

The new conditional writes feature fills a gap in distributed application patterns where you need atomic operations and coordination between multiple writers. With that in focus, clearly now the use cases can be identified. Conditional writes are implemented in couple of ways, first one is called existence check and second is called state check. S3 APIs have excellent documentaton on how the existence and state checks are implemented.

Existence Check (IfNoneMatch)

Uploads the object only if the object key name does not already exist in the bucket specified. Otherwise, operation returns an error.

State Check (IfMatch)

Uploads the object only if the ETag (entity tag) value provided during the WRITE operation matches the ETag of the object in S3. If the ETag values do not match, the operation returns an error. An ETag (Entity Tag) is a hash or version identifier that represents the state of an object, you can retrive the ETag with S3 head_object operation.

Some use cases for S3 conditional writes are

Distributed Applications In distributed systems, multiple clients or processes often need to write to shared datasets stored in S3. Without conditional writes, there's a risk of overwriting data inadvertently, leading to data loss or inconsistencies. S3 conditional writes allow these applications to implement optimistic concurrency control, ensuring that data updates occur only if the specified conditions (like a specific version or metadata) are met. This is particularly useful in environments where multiple microservices or client applications interact with the same datasets.

Large-scale Analytics Large-scale data analytics often involves parallel processing where multiple compute nodes or analytics tools access and update datasets simultaneously. For example, during map-reduce operations or iterative computations, intermediate results may be written back to S3. Using conditional writes, systems can ensure that data remains consistent across concurrent updates. This is essential for maintaining accuracy in analytics workflows and preventing corruption or duplication of processed data.

Machine Learning Workflows Machine learning pipelines frequently involve multiple processes writing model artifacts, such as checkpoints, weights, or performance metrics, to a shared S3 location. In scenarios where multiple training jobs or experiments operate simultaneously, it's crucial to ensure the integrity of shared model files. Conditional writes can prevent overwriting or corruption of critical artifacts, maintaining the integrity of the training and evaluation process. This is especially beneficial in environments with automated retraining or hyperparameter tuning workflows.

Data Pipeline Processing Data pipelines often consist of sequential processing stages, where the output of one stage becomes the input for the next. In complex workflows, multiple pipeline runs may operate concurrently, either by design or due to retries. Conditional writes enable the pipeline to validate that intermediate data isn’t overwritten or accidentally updated by a concurrent run. This ensures that each pipeline execution is isolated and maintains the integrity of processed data. For example, in ETL (Extract, Transform, Load) workflows, this can safeguard against issues arising from concurrent transformations.

Implemeting a simple conditional Write

It is time to test the conditional writes to see how to implement them from either CLI or or AWS SDK. In order to use the new functionality of conditonal writes AWS CLI needs to be version 2.22 or higher. I decided to use the aws boto3 SDK to test this. Wrote a simple python script with S3 put_object API. I am using boto3 version 1.35.74. Created two sctipts, process_1 running from one terminal window and process_2 in another terminal window. Loaded an object into my bucket - a simple text file with just one line 'Hello World'. The following snippet is used for process_1 to do put_object, with S3 exception to catch any error (not shown below)

# Get the object and its ETag process_1
        response = s3_client.get_object(
            Bucket=bucket_name,
            Key=object_key
        )
        
        original_etag = response['ETag']
        content = response['Body'].read().decode('utf-8')
        
        logger.info(f"Process 1 - Original content: {content}")
        logger.info(f"Process 1 - Original ETag: {original_etag}")
        
        # Simulate some processing time - adjust the sleep interval as needed
        time.sleep(10)
        
        # Update the object with conditional write
        new_content = f"{content} - Modified by Process 1"
        try:
            s3_client.put_object(
                Bucket=bucket_name,
                Key=object_key,
                Body=new_content,
                IfMatch=original_etag 
            )
            logger.info("Process 1 - Successfully updated the object")        

Created a separate process_2 running in a differnet terminal, which will try to update the same object

# Get the object and its ETag process_2
        response = s3_client.get_object(
            Bucket=bucket_name,
            Key=object_key
        )
        
        original_etag = response['ETag']
        content = response['Body'].read().decode('utf-8')
        
        logger.info(f"Process 2 - Original content: {content}")
        logger.info(f"Process 2 - Original ETag: {original_etag}")
        
        # Update the object immediately with conditional write
        new_content = f"{content} - Modified by Process 2"
        # Simulate some processing time - adjust the sleep interval as needed
        time.sleep(5)
        try:
            s3_client.put_object(
                Bucket=bucket_name,
                Key=object_key,
                Body=new_content,
                IfMatch=original_etag
            )
            logger.info("Process 2 - Successfully updated the object")        

As expected while process_2 is trying to update the object, process_1 attempted the update the object at the same time. Timed it such a way that first process is called from one terminal and second process from a different with a delay of 5 seconds between them. The object update by process_1 as expected failed and process_2 update succeeded. Output from process_1 and process_2 are shown below

INFO:__main__:Process 1 - Original content: Hello World
 - Modified by Process 1 - Modified by Process 2 - Modified by Process 2
INFO:__main__:Process 1 - Original ETag: "749e5960c723252862ab4a2e33190dcf"
WARNING:__main__:Process 1 - Update failed: Object was modified by another process

INFO:__main__:Process 2 - Original content: Hello World
 - Modified by Process 1 - Modified by Process 2 - Modified by Process 2 - Modified by Process 2
INFO:__main__:Process 2 - Original ETag: "c8688eba636a093ff9d7df586f8901f7"
INFO:__main__:Process 2 - Successfully updated the object        

Conclusions

S3 conditional writes is a much needed feature that bridges the gap in maintaining data consistency across distributed and concurrent workloads. Enabling optimistic concurrency control using ETags, S3 conditional writes ensure that datasets remain accurate and provide integrity, even in scenarios with multiple users or overlapping pipeline processes. It mitigates the risk of overwrites, race conditions, and data skews, empowering applications to operate with confidence in high-concurrency environments. Whether for distributed applications, machine learning workflows, or data pipeline processing, S3 conditional writes provide the reliability needed to handle shared datasets at scale. This feature is very huseful for organizations seeking to enhance data integrity while fostering seamless collaboration and efficient processing.