S3 Incremental File Processing with Pessimistic Locking using S3 Lock

What is Pessimistic Locking?

Pessimistic locking is a concurrency control mechanism that prevents multiple processes from accessing the same resource simultaneously. Unlike optimistic locking, which allows concurrent access and resolves conflicts later, pessimistic locking ensures that only one process can access a resource at a time by explicitly acquiring a lock before processing and releasing it afterward.

This approach is useful in distributed environments where multiple jobs may run concurrently, and we need to prevent data inconsistencies or conflicts.

Video Guides

Why Use Pessimistic Locking in S3 Incremental Processing?

When processing files incrementally in an Amazon S3 bucket, overlapping jobs can lead to duplicate processing or conflicts. Suppose we have a cron job that triggers every 15 minutes. If one job takes longer than 15 minutes, another instance of the job will start before the first one finishes. We need a way to ensure that only one job runs at a time.

Solution :

Example Scenario

1. Process P1 starts at 1:00 AM and acquires a lock.
2. P1 processes files incrementally and is expected to finish by 1:20 AM.
3. At 1:15 AM, Process P2 starts but sees that P1 is still running.
4. P2 waits until P1 completes and releases the lock at 1:20 AM.
5. Once P1 releases the lock, P2 acquires it and starts processing.

This ensures sequential and conflict-free execution.

Step-by-Step Guide

Step 1: Install Dependencies

First, install the required package:

Step 2: Import Required Libraries

Step 3: Implement Pessimistic Locking with S3 Lock

We use an S3-based lock to control access to the processing job.

Step 4: Implement the Worker Function

Code

https://github.com/soumilshah1995/s3-lock-file-processor/blob/main/README.md

Conclusion

Using pessimistic locking with S3, we can prevent overlapping jobs from causing data inconsistencies. The first job to acquire the lock processes files, while any overlapping jobs wait until the lock is released. This ensures a sequential, reliable processing pipeline.

By implementing this approach, you can avoid conflicts and efficiently manage incremental file processing in AWS S3.

#AWS #S3 #DataEngineering #PessimisticLocking #ConcurrencyControl