Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

WHAT IS LVM ?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.

With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks.

The physical volumes are combined into logical volumes, with the exception of the /boot partition. The /boot partition cannot be on a logical volume group because the boot loader cannot read it. If the root (/) partition is on a logical volume, create a separate /boot partition which is not a part of a volume group.

Since a physical volume cannot span over multiple drives, to span over more than one drive, create one or more physical volumes per drive.

PRE-REQUISITES

Hadoop cluster configured with one Name Node and one Data Node.

Knowledge of LVM.

PROCEDURE

Step 1: Create the cluster with one data node and master node. Initially we have attached 10GB Hard disk and provided to Name Node.

Step 2: Attach external hard disk to the Data Node. We can see that we have attached 2GB extra to the Data Node.

Step 3: Checking PV . Initially no PV found.

#lsblk

Step 4: Now we will create the PV (Physical Volume)

    #pvcreate /dev/xvdg

PV successfully created.

Step 5: Now we will create the VG (Volume Group)

    #vgcreate arth_task_7.1_vg /dev/xvdg
    #vgdisplay

VG created successfully we can check using vgdisplay command.

Step 6: Now we will create LV (Logical Volume)

   #lvcreate --size 500M --name lv7.1 arth_task_7.1_vg
   #lvdisplay

LV is successfully created. We can confirm using above command.

Step 7: Format the LV using mkfs.ext4 

Also create a directory to use the LV.

Step 8: Create the directory and mount the LV to that directory

Step 9: Update the hdfs_site.xml with LV name in Data Node

Now we can see by running Hadoop dfs-admin -report command that the cluster is getting the storage from the created LV.

Here we can see that the size of LV is allocated to the cluster.

Step 10: Now we will extend the LVM partition using lvxtend command.

Also we have to use resize2fs command to deflect the changes storage of LV

Now again if we run Hadoop dfs-admin command we can see that the storage is increased on the fly.

Hence we have successfully attached the LV and provided elasticity to the Hadoop Cluster.

Thank You.