Learn How to Perform Dual Write: S3 Table Buckets and Unmanaged Iceberg on EMR EC2, and Sync with AWS Glue | Required Configuration

Introduction

Managing large-scale data lakes efficiently requires advanced techniques like dual write, where data is simultaneously written to two different table formats. In this guide, we will demonstrate how to perform dual writes into Amazon S3 Table Buckets and Unmanaged Apache Iceberg tables on an EMR EC2 cluster, ensuring synchronization with AWS Glue.

By the end of this blog, you’ll understand:

• How to configure Spark for writing to both S3 Table Buckets and Unmanaged Iceberg.
• How to set up and manage catalogs in AWS Glue.
• When to use dual write and why it is beneficial.

Video Guide

Why Dual Write?

Use Cases

Migration to a New Service

• If migrating to a newer pipeline, dual writes allow you to maintain both old and new formats.
• Once the new pipeline is validated, the older one can be retired safely.

Performance Evaluation

• Comparing S3 Table Buckets and Unmanaged Iceberg helps in understanding query performance and storage efficiency.

Spark Submit Job

Understanding Catalog Configurations

Unmanaged Iceberg Catalog Configuration

• Uses AWS Glue as the catalog implementation for metadata management.
• S3FileIO is used to handle object storage interactions.

Managed Iceberg Catalog Configuration (S3 Table Buckets)

• Uses AWS Glue as the catalog implementation for metadata management.
• S3FileIO is used to handle object storage interactions.

Writing Data to Both Tables

Spark Script for Dual Write

Code https://github.com/soumilshah1995/dual-write-iceberg-s3

Conclusion

By setting up a dual write architecture, you can migrate workloads, evaluate performance, and ensure seamless synchronization with AWS Glue. With Iceberg’s powerful features and AWS’s managed capabilities, you get the best of both performance and metadata management.

Happy coding!

Follow me

Linkedin | Blog | Youtube | Medium | Github | Instagram | Website