Developer Guide: How to Submit Hudi PySpark(Python) Jobs to EMR Serverless (7.1.0) with AWS Glue Hive MetaStore

Apache Hudi is a powerful data management framework that brings ACID transactions to data lakes. Running Hudi jobs on Amazon EMR Serverless combines the scalability of EMR with the flexibility of serverless computing. This step-by-step guide will walk you through submitting a Hudi PySpark job to EMR Serverless.

Video Guides

Step 1: Create an EMR Serverless Application

Before running any jobs, you need to create an EMR Serverless application. This is a one-time setup step:

1. Open the AWS Management Console and navigate to EMR Serverless.
2. Create a new application:Select Spark as the framework.Define your application name.Configure network settings and logging (e.g., S3 bucket for logs).

Click on Create Cluster

Verify the cluster and copy application ID

Step 2: Write Your Hudi PySpark Job

Upload the File to S3

Export ENV variables

Submit Job

Output

Query Athena

Exercise Files

https://github.com/soumilshah1995/emr-hudi-getting-started/blob/main/README.md

Conclusion

Running Hudi jobs on EMR Serverless simplifies the management of large-scale data lakes by leveraging serverless computing. By following this guide, you’ve set up and run a Hudi PySpark job on EMR Serverless, taking advantage of both the flexibility of PySpark and the robustness of Hudi.

Feel free to customize the script and explore more Hudi functionalities to suit your data processing needs.