Integrating LVM with Hadoop and providing Elasticity to Data Node Storage

What is LVM?

In Linux, Logical Volume Manager is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

What is Elasticity in terms of Storage?

It refers to the capability of a storage system to adapt to variable workload changes by allocating and deallocating resources as required by each application.

What is Static or Fixed Partition?

This is the oldest and simplest technique used to put more than one processes in the main memory. In this partitioning, number of partitions (non-overlapping) in RAM are fixed but size of each partition may or may not be same. As it is contiguous allocation, hence no spanning is allowed. Here partition are made before execution or during system configure.

Problem Statement

?? 7.1: Elasticity Task

7.1.1 Integrating LVM with Hadoop and providing Elasticity to Data Node Storage

7.1.2 Increase or Decrease the Size of Static Partition in Linux.

Solution Steps

Task 7.1.1

Step 1:- CREATE PHYSICAL VOLUME. From the extra hard disks that you have attached.

Step 2:- CREATE VOLUME GROUP

Step 3:- CREATE LOGICAL VOLUME-

Step 4:- SHOW ALL THE LOGICAL VOLUME OF MY OS-

Step 5:- FORMAT THE LOGICAL PARTITION-

Step 6:- MOUNT

Partition is stored in the location /dev/mapper

Step 7:- Increase the size of LV

Use resize2fs command to format the extended portion of the LV

LVDISPLAY

Step 8:- NOW WE HAVE TO INTEGRATE WITH HADOOP. FOR THAT WE NEED TO CONFIGURE THE SYSTEM WHERE WE MADE A LOGICAL VOLUME AS A DATANODE , AND THE OTHER AS A MASTER.

HERE I HAVE CONFIGURED THE MASTER IN ANOTHER VM BY EDITING THE CONFIGURATION FILES: - hdfs-site.xml and core-site.xml , present in the /etc/hadoop directory.

Step 9:- CONFIGURE THE DATE NODE SIMILARLY

HERE THE DIRECTORY IN THE CONFIGURATION SHOULD BE SAME IN WHICH WE MOUNTED THE LVM , THAT IS /myfold.

Step 10:- FORMAT THE NAME NODE AND START IT

Step 11:- START THE DATA NODE

Step 12:- CHECK THE CLUSTER STATUS IN THE NAME NODE, WHERE WE WILL SEE THE CONTRIBUTED STORAGE OF DATA NODE TO THE CLUSTER.

Step 13:- NOW WE WILL INCREASE THE SIZE OF LVM AND CHECK IF IT IS REFLECTED IN THE HADOOP CLUSTER DETAILS.

THE ABOVE PICTURE CLEARLY DEPICTS THAT AS WE INCREASED THE SIZE OF LVM IN THE DATA NODE, ITS CONTRIBUTED STORAGE TO THE CLUSTER ALSO INCREASES FROM 16 GB(approx.) TO 18 GB(approx.).

HENCE , INTEGRATION OF HADOOP AND LVM WAS SUCCESSFUL.

*******************************************************************************

SOLUTION STEPS

TASK 7.1.2

We can see available disks using command : #fdisk -l

Step 1:- Create static Partition of attached disk.

In my case I will create first partition of size 5GB on disk named /dev/sdb

steps for creating partition.

1. fdisk /dev/sdb: to go inside the disk
2. n : for new partition
3. p : for primary partition
4. w: to save the created partition.

Step 2:- Now format and mount the partition.

FORMAT

Then we will mount that formatted partition on the directory named /geetansh

Step 3:- Now we will create file in the created folder.

In this case , I have created file named spartition.txt inside the folder named geetansh.

Now let’s see how to increase the static partition

Step 4:- Increase the size of static partition

Unmount the partition

Delete the partition

d : it is for deleting partition

w : to save the changes

Step 5:- Create a new partition , format it and mount it on the directory geetansh

Create new partition

create new partition of size 7GB. We already formatted the partition1, now it is asking to remove or save the ext4 signature , in my case I am not removing the ext4 signature.

Here “N” is for don’t remove the ext4 signature.

and finally save the changes using #w

Now partition of 7gb is created.

Now format the extended part of partition.

1. Examine filesystem for errors

2. Format the extended part :

Again Mount the created partitions on the same folder which we created previously

and after mounting partition check by using #df -h whether size has increased or not

In my case partition has mounted of size nearly about 7GB

Step 6:- Check if previously created file i.e. spartition.txt exists or not in the /geetansh directory.

Hence, we have increased the size of previously created static partitions without losing any data.

************************THANKS FOR READING**************************