HADOOP HDFS

In this article, we'll dive deeper into the Hadoop Distributed File System (HDFS), focusing on its intricate mechanisms. We'll explore its write and read operations, data storage architecture, data pipeline, and fault tolerance features. Understanding these technical aspects is crucial for effectively utilizing HDFS in large-scale data processing and analytics tasks. Let's unravel the inner workings of HDFS and how it ensures efficient and reliable management of massive datasets in distributed computing environments.

Read Mechanism in HDFS

The read process in HDFS involves several steps to retrieve data stored across multiple nodes:

1. Client Request: The client interacts with the HDFS client library to read a file.
2. Namenode Interaction: The client requests the namenode to fetch the metadata of the file, which includes the block locations.
3. Block Location: The namenode responds with the block locations and the datanodes that store the replicas of these blocks.
4. Data Retrieval: The client connects to the closest datanode to start reading the data blocks.
5. Sequential Read: If the file is large and spans multiple blocks, the client continues to read from different datanodes as specified by the namenode.

Detailed Steps of Read Operation:

1. Open File: The client calls the open() method on the FileSystem object to get the FSDataInputStream.
2. Request Metadata: The client sends an open request to the namenode, which includes the filename.
3. Namenode Response: The namenode looks up the metadata for the file and returns the block locations (list of blocks and datanodes).
4. Read Blocks:

• Block 1: The client connects to the first datanode that holds the first block and reads the data.
• Block 2: After finishing Block 1, the client connects to the datanode holding the second block and reads the data.

1. Error Handling: If a datanode fails, the client connects to another datanode holding a replica of the block.
2. Checksum Verification: The client verifies checksums to ensure data integrity. If a mismatch is detected, it reads from another replica.
3. Complete Read: The process continues until the client reads all blocks of the file.

Write Mechanism in HDFS

The write process in HDFS is designed to ensure data reliability and integrity:

1. Client Request: The client requests the namenode to create a new file.
2. Namenode Response: The namenode checks for file existence (to avoid duplicates) and assigns a block ID.
3. Data Block Division: The client splits the data into smaller chunks, typically 128 MB or 256 MB.
4. Block Assignment: The namenode assigns a list of datanodes for each block, which will store the replicas.
5. Write Pipeline: The client writes the data to the first datanode, which then forwards it to the next datanode, forming a pipeline until the final datanode in the replication chain.
6. Acknowledgment: Each datanode sends an acknowledgment back to the previous node and ultimately to the client.

Detailed Steps of Write Operation:

1. Create File: The client calls the create() method on the FileSystem object to create a new file in HDFS.
2. Request Block Locations: The client sends a request to the namenode to create the file and obtain block locations.
3. Namenode Assigns Blocks: The namenode assigns blocks and provides the list of datanodes for replication.
4. Write to Pipeline:

• Pipeline Setup: The client streams the first block to the first datanode in the pipeline.
• Data Transfer: The first datanode writes the block to its local storage and forwards the block to the second datanode.
• Replication: The second datanode writes the block to its local storage and forwards it to the third datanode, completing the replication process.

5. Acknowledgment Process:

• Block Storage: Each datanode stores the block and sends an acknowledgment back through the pipeline.
• Client Confirmation: The client receives the final acknowledgment, confirming the block's successful write.

6. Next Block: The client proceeds to the next block and repeats the process until the entire file is written.

Write Pipeline and Acknowledgment:

• Write Pipeline: The client writes the data to the first datanode, which then forwards it to the second datanode, and so on, creating a pipeline.
• Acknowledgment: Each datanode sends an acknowledgment back to the previous node and ultimately to the client once the data block is successfully written and replicated.

Example Sequence for Read:

1. Client requests to read file.txt.
2. Namenode responds with metadata: Block1 (Datanode1, Datanode2), Block2 (Datanode3, Datanode4).
3. Client connects to Datanode1 to read Block1.
4. Client reads Block1 and verifies checksum.
5. Client connects to Datanode3 to read Block2.
6. Client reads Block2 and verifies checksum.
7. Client assembles the complete file data from the blocks.

Example Sequence for Write:

1. Client requests to write file.txt.
2. Namenode assigns Block1 to Datanode1, Datanode2, and Datanode3.
3. Client writes Block1 to Datanode1.
4. Datanode1 stores Block1 and forwards it to Datanode2.
5. Datanode2 stores Block1 and forwards it to Datanode3.
6. Datanode3 stores Block1.
7. Acknowledgments flow back from Datanode3 to Datanode2, from Datanode2 to Datanode1, and finally from Datanode1 to the client.
8. Client proceeds with Block2 and repeats the process.

This read and write mechanism ensures that HDFS provides reliable, scalable, and efficient access to large datasets in a distributed computing environment.

Data Block in HDFS

HDFS stores files by dividing them into blocks:

• Default Block Size: By default, a data block in HDFS is 128 MB or 256 MB, but this can be configured.
• Fixed Size: Each file is split into fixed-size blocks, which simplifies storage management and helps in handling large files efficiently.
• Distributed Storage: These blocks are distributed across the nodes in a Hadoop cluster.
• Parallel Processing: Blocks can be processed in parallel across different nodes, enhancing performance and speed.

Fault Tolerance and Replication in HDFS

HDFS ensures data reliability and fault tolerance through replication:

• Replication Factor: Each block of data is replicated across multiple datanodes (default replication factor is 3).
• Rack Awareness: Replicas are stored on different racks to ensure data availability even in case of a rack failure. Typically, one replica is stored on a different rack to avoid single points of failure.
• Heartbeat and Block Reports: Datanodes send periodic heartbeats and block reports to the namenode to confirm their status and the blocks they are storing.
• Re-replication: If a datanode fails, the namenode detects the missing blocks and initiates replication to maintain the specified replication factor.

Write Pipeline in HDFS

The write pipeline ensures efficient data distribution and replication:

1. Client Initiates Write: The client initiates the write operation by contacting the namenode.
2. Pipeline Formation: The namenode returns the datanodes that will store the block replicas. These datanodes form a pipeline.
3. Data Streaming: The client streams the data to the first datanode in the pipeline.
4. Pipeline Forwarding:

• First Datanode: The first datanode stores the block and forwards the data to the second datanode in the pipeline.
• Second Datanode: The second datanode receives the data, stores the block, and forwards it to the third datanode.
• Third Datanode: The third datanode receives the data and stores the block, completing the replication process.

Acknowledgment in Write Pipeline

Acknowledgments ensure data integrity and successful writes:

1. Sequential Acknowledgment: After a datanode stores a block, it sends an acknowledgment back to the previous datanode in the pipeline.
2. Client Acknowledgment: The acknowledgment travels back through the pipeline from the last datanode to the client.
3. Success Confirmation: When the client receives acknowledgments from all the datanodes in the pipeline, it considers the write operation successful.
4. Error Handling: If any datanode fails to send an acknowledgment, the client retries the write operation or the namenode chooses new datanodes to complete the replication.

Detailed Steps of Write Operation:

1. Client Requests to Create a File: The client calls the create() method on the FileSystem object to create a new file in HDFS.
2. Namenode Assigns Block Locations: The client sends a request to the namenode to create the file and obtain block locations.
3. Namenode Assigns Blocks: The namenode allocates blocks and provides the list of datanodes for replication.
4. Data Streaming and Replication:

• Client Writes to First Datanode: The client streams the block to the first datanode.
• Pipeline Forwarding: The first datanode writes the block to its local storage and forwards the block to the second datanode.
• Second Datanode: The second datanode writes the block to its local storage and forwards it to the third datanode.
• Third Datanode: The third datanode writes the block to its local storage, completing the replication process.

5. Acknowledgment Process:

• Block Storage: Each datanode sends an acknowledgment back to the previous datanode after storing the block.
• Client Confirmation: The client receives the final acknowledgment, confirming the block's successful write.

6. Next Block: The client proceeds to the next block and repeats the process until the entire file is written.

Example Sequence for Write:

1. Client Request: Client requests to write file.txt.
2. Namenode Assignment: Namenode assigns Block1 to Datanode1, Datanode2, and Datanode3.
3. Client Writes Block1: Client writes Block1 to Datanode1.
4. Pipeline Forwarding:

• First Datanode: Datanode1 stores Block1 and forwards it to Datanode2.
• Second Datanode: Datanode2 stores Block1 and forwards it to Datanode3.
• Third Datanode: Datanode3 stores Block1.

1. Acknowledgment Process:

• Datanode to Client: Acknowledgments flow back from Datanode3 to Datanode2, from Datanode2 to Datanode1, and finally from Datanode1 to the client.

1. Completion: Client proceeds with Block2 and repeats the process.

This write mechanism ensures that data is reliably written to HDFS with proper replication, providing high availability and fault tolerance.