Learn How to use Hudi DeltaStreamer with Hudi 0.14 on AWS Glue: A Seamless Data Ingestion Guide

In today's data-driven world, the ability to efficiently ingest and process data from various sources is paramount for businesses aiming to stay competitive. Apache Hudi (Hadoop Upserts Deletes and Incrementals) has emerged as a powerful tool for managing large datasets with incremental data ingestion capabilities. With the release of Hudi 0.14, coupled with AWS Glue's robust data transformation and ingestion capabilities, leveraging the Hudi DeltaStreamer becomes even more seamless.

Video Guides

Understanding the Power of DeltaStreamer

DeltaStreamer, a key component of Apache Hudi, simplifies the process of ingesting data into Hudi tables. Its ability to efficiently handle real-time data streams and perform upserts, deletes, and incremental data ingestion makes it a valuable asset for data engineers and architects. By leveraging DeltaStreamer, organizations can streamline their data pipelines and ensure data consistency and reliability.

Setting Up the Environment

Before diving into the intricacies of using Hudi DeltaStreamer with AWS Glue, let's ensure our environment is set up correctly. Follow these steps to get started:

Step 1: Download Dataset and Upload to S3

Grab the dataset from the provided link and upload it to your desired S3 bucket. This dataset will serve as our sample data for ingestion.

https://drive.google.com/drive/folders/1BwNEK649hErbsWcYLZhqCWnaXFX3mIsg?usp=share_link

Step 2: Download the Required Jar Files

Ensure you have the necessary JAR files, including

jcommander-1.78.jar

hudi-spark3.3-bundle_2.12-0.14.0.jar

hudi-utilities-slim-bundle_2.12-0.14.0.jar.

These files are crucial for running Hudi DeltaStreamer

Download Jar

https://drive.google.com/drive/folders/1Rs9243i-D-jmFHPivlwdtODBRQpG1nys?usp=share_link

Step 3: Upload the Code to AWS Glue

Use the provided Scala code snippet to upload the necessary code to AWS Glue. This code initializes the DeltaStreamer and orchestrates the ingestion process seamlessly.

Make sure to add Jar Path in Job config

Ingesting Data with Hudi DeltaStreamer on AWS Glue

The provided Scala code snippet initializes the DeltaStreamer and configures it to ingest data from a specified source to an Apache Hudi table. Here's a brief overview of the key components:

• Source Configuration: Specify the source class and ordering field to ensure data consistency during ingestion.
• Target Configuration: Define the target base path, table name, and table type (e.g., COPY_ON_WRITE).
• Hudi Configuration: Set up Hudi-specific configurations, such as key generator, record key field, partition path field, and source DFS root.
• Spark Context Initialization: Initialize the Spark context for executing the data ingestion job.
• Glue Context Setup: Set up the Glue context for seamless integration with AWS Glue services.
• DeltaStreamer Execution: Execute the DeltaStreamer to synchronize the data from the source to the target Hudi table.

GH: https://github.com/soumilshah1995/deltastreamer-on-glue

Conclusion

Incorporating Hudi DeltaStreamer with Hudi 0.14 on AWS Glue provides a streamlined approach to ingesting and managing large datasets. By following the steps outlined in this guide, data engineers and architects can harness the power of DeltaStreamer to ensure efficient and reliable data ingestion from various sources. Embrace the capabilities of Apache Hudi and AWS Glue to propel your data pipelines towards success.

References

• AWS Blog: Ingest Streaming Data to Apache Hudi Tables Using AWS Glue and Apache Hudi DeltaStreamer
• LinkedIn Article: Hudi Streamer Delta - Hands-on Guide for Local Ingestion
• Apache Hudi Documentation: Hoodie Streaming Ingestion

With the seamless integration of Hudi DeltaStreamer with AWS Glue, the possibilities for efficient data management and analytics are endless. Start leveraging these powerful tools today to unlock the full potential of your data infrastructure. Happy ingesting!

Special Thanks

Thank you Aditya Goenkar for all your help

Note:

If you're considering utilizing Hudi DeltaStreamer via the Glue connector, it's worth noting that an AWS Blogs post referenced an older version of Hudi. For a smoother experience, I recommend following a guide that incorporates the latest version of Hudi.

Want to Master Apache Hudi DeltaStreamer ?

Complete 15 + Videos with all Hands on Guide all for free

https://www.youtube.com/watch?v=s42-mGktIpg&list=PLL2hlSFBmWwz2lp7K8dMk8hWV1p_SNeBQ