Concurrent Writes Test for New S3 Table Buckets: Can 10 Spark Writers Performing MERGE INTO Different Partitions Handle It?
Introduction
In modern big data applications, managing concurrent writes to distributed storage systems like Amazon S3 is a critical challenge. When dealing with large volumes of data, it’s often necessary to orchestrate multiple concurrent Spark writers, each performing operations like MERGE INTO across different partitions in the data lake. In this blog, we'll walk through how to manage these concurrent writes efficiently, using Apache Iceberg with Apache Spark to handle the complexities of partitioned data. We’ll also discuss how to configure key settings in Iceberg to ensure successful concurrent writes and provide a working example for testing these configurations.
The Challenge: Concurrent Writes to S3
When dealing with concurrent writes, especially with multiple Spark writers performing MERGE INTO operations on partitioned tables, there are challenges related to consistency, data integrity, and performance. Iceberg’s configuration provides robust options to manage these challenges.
In this example, we simulate 10 Spark writers running in parallel, each writing to different partitions of an Iceberg table stored in S3. We'll use the MERGE INTO SQL command to update records across different partitions.
Key Configurations in Iceberg
To handle concurrent writes efficiently, we rely on the following Iceberg configurations:
This configuration specifies the number of retries for a commit if a write fails. Setting this value ensures that Iceberg can automatically retry committing the transaction a number of times if conflicts or transient issues occur, thus ensuring reliability.
The minimum wait time (in milliseconds) between retries. This setting helps avoid overwhelming the system by spacing out retries.
This configuration controls the isolation level of the MERGE INTO operations. Setting it to snapshot ensures that the operation works on a snapshot of the data, providing a consistent view for concurrent writers, preventing conflicts during the write process.
Example: Test Case Setup
Let’s walk through an example where we simulate concurrent writes using 10 Spark writers, each performing MERGE INTO operations across different partitions.
Step 1: Create Iceberg table
Spark Writer Job
PySpark Code
Lets Submit 10 Spark jobs
Final Output
Conclusion
All MERGE INTO commands executed successfully on the Iceberg table, and although there were some errors during the process, the retry settings ensured that the operations were eventually successful. The commit.retry.num-retries, commit.retry.min-wait-ms, and write.merge.isolation-level configurations played a key role in guaranteeing reliable and consistent writes. These settings allowed for retries in case of failures, ensuring that all data writes successfully made it into the new S3 table buckets without losing any records.