Limiting The Storage In Hadoop Cluster By Data Node

!! ?????????? ?????????????????????? !!

? Welcome you all to my article based on ???????? - ??.?? of ???????? - ?????? ???????????? ???? ?????????????????????????

?? TASK DESCRIPTION:

?? In Hadoop cluster , find how to contribute limited/specific amount of storage as a slave node to the cluster ?

Hint : Use Of Partitions

?? TASK COMPLETION:

1) I have build the Hadoop cluster on AWS Cloud .So , Let's start the Hadoop services i.e Name Node and Data Node .

2) As we store any folder or directory inside one storage drive like for example C-drive .Normally by default the folder takes total size of that storage drive . In case of Hadoop cluster we give storage of Data Node by some folder or directory which by default shares all the storage amount of the drive inside which the folder is present .

We can check the storage amount of the Drive inside which the Data Node storage directory is present by using command :

df -h

These Linux Command shows all the Information of amount of available and used Hard disk or volume space .

We can see that the size of / drive is 10 GB . As In my case the Data Node Directory is mounted on / drive . Let's check that Data node shares the total storage of / drive .

hadoop dfsadmin -report

3) Now we have to find some way by which the we can control or limit these Data Node storage in Hadoop cluster . For these we have to use the concept of disk partitions .So , Lets create one EBS Volume of some size and attach it to the Data Node .

We can see all the disk partitions present in our Data Node using Linux Command :

fdisk -l

4) We have attached the EBS Volume /dev/xvdf of 15 GB to the Data Node . To create the partitions we have command as :

fdisk  device_name

We just have to specify the amount or size of Last Sector of primary partition . In my case I have created the partition of size 8 GB .

5) Let's check the partition is successfully created or not . We see that one partition named as /dev/xvdf1 is created of size 8 GB .

6) Now we have to format the partition . To format the partition we have command as :

mkfs.ext4  device_name

7) After formatting the partition we have to mount our Data Node directory to the new partition we have created . To mount the partition we have command as :

mount  device_name  directory_name

8) We can see that now we have control or Limit on Data Node Storage as /dn1 Data Node Directory is now access the storage that we have provided to it .

? We see that now the Data Node not takes the total storage of EBS Volume of size 15 GB as we have control or provided it specific amount of storage using the concept of disk partitions .

?? In these way I successfully completed the  ???????? - ??.?? of ARTH - The School Of Technologies .

? I would like to thanks Mr.Vimal Daga for giving such research based task which helps me to explore my core concepts of Big Data - Hadoop .

??For any queries or suggestions DM me .

!! Thanking you all for visiting my article !!

?? Keep Sharing Keep Learning ??