How to contribute limited amount of storage as slave to the cluster

Task Description:-

In a Hadoop cluster, find how to contribute limited/ specific amount of storage as slave to the cluster ?

Hint:- Linux partitions

Prerequisite:- Configure one Datanode and one Masternode.

We have to perform following steps :-

1. Create one hard disk and attached it to the data node.
2. Create partition.
3. Again configure the data node.
4. Check storage shared by data node to name node.

Implementation:-

Step 1:- Create one hard disk.

Now I am going to check it through terminal. So open the terminal of the data node and run this command which is given below.

Command:-

fdisk -l

Here we can see that the hard disk is attached to the data node.

First step is done successfully now move to the step2.

Step 2:- Create partition:-

For creating partition run this command which given below.

Command:-

fdisk /dev/xvdf

Here I am going to use some options.

1. Create new partition:- n
2. Primary partition:- p
3. Partition number:- 1(by default)
4. First sector:- 2048(by default)
5. Last sector:- 1G (it's depend on you how much space you want in my case it's 1G)
6. Save this partition:- w

Here our partition is done successfully.

Format the created partition

There are so many ways for formatting the partition I am going to use ext4 and after mentioning the format type we have to tell which partition we want to format and command to format the partition is given below.

command:-

mkfs.ext4 /dev/xvdf

Mount that partition

For mounting the partition we have to first create directory and the mount it to that directory. Run the command given below.

command to create directory:-

mkdir /datanode

command to mount it:-

mount /dev/xvdf  /datanode

Here the step 2 is successfully done now move to the step 3

Step 3:- Configure the data node.

We have to just configure the hdfs-site.xml file of data node and give the directory name which we are created above (in my case it's /datanode)

Here step 3 is successfully done now we have to move to the step 4.

Step 4:- Check storage shared by data node to the master node.

Now we have to check that how much storage is shared. So run one command either on data node or master node and command is given below.

command:-

hadoop dfsadmin -report

Here we can see that the configured capacity is 975.9 MB, which is almost 1 GB and in this way we can contribute specific amount of storage as slave to the cluster.

Here my task is completed successfully.