Task 7.1 ??????

Task Description??

?? Elasticity Task

??Integrate LVM with Hadoop and provide Elasticity to DataNode Storage

??Increase or Decrease the Size of Static Partition in Linux.

??Automating LVM Partition using Python-Script.

This is an important article on Hadoop and LVM. In this article,?I have launched and configured 2 instances one for Hadoop master(Name node) and the other for Hadoop slave(Data node).

So, here we start with our task:-

Step -1: Creation of two instances on AWS Cloud one for Master Node and the other for Slave node.

Step -2: Download and install JDK software in both instances.

These are the following software which we have to download and install:

jdk-8u171-linux-x64.rpm

hadoop-1.2.1–1.x86_64.rpm

Since I already have this software in my local machine I will transfer it to instances by using the WinSCP tool.

Step -3: Installing the Software in Master and slave node.

Let's check the version and

Step -4: Configuring name node by updating hdfs-site.xml and core-site.xml file

hdfs-site.xml (master node)

core-site.xml (master node)

Step -5: Format the name node

when prompt to reformat filesystem press Y

Step -6: Use hadoop-daemon.sh start namenode command to start name node.

Step -7: Use jps command to?check if a specific daemon is up or not.

YEAH!!! IT'S WORKING

Step -8: Use hadoop dfsadmin -report command to see whether the slave node is attached or not.

Here it is not attached. so now we will configure the instance for the slave node.

Step -9: Configuring the instance for the slave node.

hdfs-site.xml (slave node) and core-site.xml (slave node)

Use hadoop-daemon.sh start datanode command to start name node.

Use jps command to?check if a specific daemon is up or not.

Use hadoop dfsadmin -report command to see whether the slave node is attached or not.

Here you can see slave node is attached to the master node.

Step -10:Create 2 EBS volumes and attach them to slave (data node)

Step - 11: fdisk is used for the view, create, delete, change, resize, copy and move partitions on a hard drive using the dialog-driven interface.

Step-11: If pvcreate cmd is not available then download it

Download and Install lvm2

lvm2 downloaded

Step-12: Creation of Physical volumes.

pvdisplay cmd to see the details.

Step-13: Creation of Volume group.

Step-13: Creation of Logical Volume.

Step-14: Formatting the partition

Step-15: Mount the partition to the slave node folder

Step-16: df -h command to defines the number of blocks used, the number of blocks available, and the directory where the file system is mounted.

The size is now increased from 3.9 Gib to 5.9 Gib approx.

As you can see here we have 3.87 Gib in configured capacity after increasing it by +2G it will become

5.84 Gib in configured capacity.

Step-17: Some of the screenshots of the Python script for LVM

Link for LVM Script: https://github.com/Pratikkohad1999/Automating-LVM-Partition-using-Python-Script..git

Thanks for reading the article.

Hope this article is helpful to you!!!!

You can appreciate the article by giving it a like and posting comments about your feedback.

Feel free to ask any queries. Don’t forget to share this article with your colleagues.??