Controlling Hadoop Storage

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Big Data :- Big data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques and frameworks.

Hadoop :- Hadoop is an open source, Java based framework used for storing and processing big data. The data is stored on inexpensive commodity servers that run as clusters. Hadoop uses the MapReduce programming model for faster storage and retrieval of data from its nodes.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

In a Hadoop Cluster, a limited/specific amount of storage can be contributed. This can easily be initiated with the help of partitions in the following ways:-

• Create a Hadoop cluster containing Name Node and a Data Node

• The Data Node in this case with 10GB of storage is mounted on / drive contributing all of its storage to the Master Node. For controlling the storage attach an EBS volume of desired size with the Data Node.

• All the disks present in our Data Node can be viewed as follows.

lsblk  or  fdisk -l

• An EBS volume /dev/xvdf of 20GB has been attached to the Data Node. Now, its time to create a physical partition of desired size. I have created a partition of size 2GB.

fdisk /dev/xvdf

• The next step is formatting the partition being created.

mkfs.ext4 /dev/xvdf1

• The final step , mount the partition to our Data Node directory.

mount /dev/xvdf1 /dn1

• The limited storage of 2GB being shared can now be viewed easily.

THANK YOU