Create EMR Transient Cluster, Submit PySpark Job with Async Callback, and Auto-Terminate the Cluster

n this blog, we'll walk through creating and managing an EMR (Elastic MapReduce) cluster on EC2 to run PySpark jobs using AWS Step Functions. We will employ an asynchronous callback pattern, ensuring that the cluster is terminated automatically after the job completes.

Overview

The steps involved include:

1. Creating a Step Function to manage the lifecycle of an EMR cluster.
2. Uploading necessary files, including bootstrap scripts and PySpark job scripts, to S3.
3. Configuring the cluster and running the job.
4. Automatically terminating the EMR cluster upon job completion.

Video Guides

Step 1: Create a Step Function

Below is the JSON definition for the Step Function:

Code to set function JSON: LINK

Step 2: Upload Files to S3

To prepare for the EMR job, upload the necessary bootstrap and job scripts to an S3 bucket.

Bootstrap Script

Below is a sample bootstrap script to install dependencies on the cluster:

Sample PySpark Script

Upload both of these scripts on S3

Step 3: Verify S3 Files

Ensure that the files have been successfully uploaded to your S3 bucket. Navigate to the AWS S3 Console and check the scripts directory for the bootstrap.sh and test_job.py files.

Step 4: Submit the EMR Job

Here is a sample payload for submitting the EMR job through Step Functions:

Conclusion

By leveraging AWS Step Functions to manage EMR clusters, you can automate the lifecycle of your PySpark jobs on AWS. The setup ensures that resources are utilized efficiently, with the cluster being terminated as soon as the job completes, reducing costs and operational overhead.