Learn How to configure Trino with Hudi and Hive Metastore with MINIO Object Store Developer Guide

In the realm of big data processing, efficient data storage and querying are paramount. Technologies like Trino , Apache Hudi, and Hive Metastore play pivotal roles in achieving seamless data handling at scale. In this guide, we'll walk through the process of configuring Trino with Hudi and Hive Metastore while leveraging MINIO Object Store for storage.

Video Guide

https://www.youtube.com/watch?v=gfU5_WEX1cM&feature=youtu.be

Step 1: Setting up the Environment

We'll begin by defining our environment using Docker Compose. Below is a sample docker-compose.yml file:

Step 2: Configuring Trino and Hudi

After setting up the environment, we need to configure Trino, Hudi, and Hive Metastore. Here are the configuration files and their explanations:

trino/etc/node.properties

This file specifies Trino node properties, including environment setup, data directory, and plugin directory.

trino/etc/jvm.config

This configures JVM options for Trino, optimizing memory usage and garbage collection.

trino/etc/config.properties

This sets up Trino as a coordinator node, specifies HTTP server port, and enables service discovery.

trino/etc/catalog/hudi.properties

This configures the Hudi connector with the Hive Metastore URI, MINIO Object Store credentials, and endpoint details.

Step 3: Sample Code Execution

create spark session

Use following Hudi property to do hive sync

Write data into Hudi

Query Via Trino

Output:

These queries demonstrate how to connect to Trino and execute SQL commands to interact with the data stored in Hudi.

With these steps, you've successfully configured Trino with Hudi and Hive Metastore using MINIO Object Store, enabling seamless big data processing and querying capabilities.

GH Link https://github.com/soumilshah1995?tab=repositories

Note: It's important to mention that the newer version comes with significant changes, including updates to the Java version. However, it's worth noting that this update has introduced a few bugs when querying data via Trino Hudi. Therefore, it's recommended to stick with the lower version until these issues are resolved.