Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
Anudeep Nalla
Opensource Contributer | Platform Engineer | EX-NPCI | RHCA Level III | OpenShift | CEPH | CK{S,A,AD} | 3x Microsoft Certified | AWS CSA | Rancher | Nirmata | DevOps | Ansible | Jenkins | DevSecOps | Kyverno | Rook-Ceph
?? 7.1: Elasticity Task
??Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
??Increase or Decrease the Size of Static Partition in Linux.
??Automating LVM Partition using Python-Script.
step1: I have created 2 hard disks. Of 25Gib and 50Gib
Check the hard disks using
fdisk -l
Step 2: Create the Physical Volume for both the hard disks using
pvcreate /dev/sdc pvcreate /dev/sdb
To check the physical volume is created or not using
pvdisplay /dev/sdb pvdisplay /dev/sdc
Step3: Create the volumegroup (vg) using
vgcreate <vg_name> <HD1_name> <HD2_name> vgcreate lvmhadoop /dev/sdb /dev/sdc
Step4: Create the logical volume with size of 10Gi from lvmhadoop volumegroup(vg) using
lvcreate --size 10G --name <LV_name> <VG_name> lvcreate --size 10G --name mylv1 lvmhadoop
Format the logical volume using mkfs.ext4 /dev/lvmhadoop/mylv1
Make the new directory and mount the logical volume to that directory
Step5: Start the services of Hadoop namenode
Step6: Start the services of Hadoop NameNode
Step7: Now I want to increase the lv from 10Gi to 30Gi
After extending the lv to 30Gi, we must format it too. In this case, if we use mkfs.ext4, the data will be lost. So, we must use resize2fs.
Step8: To reduce the lv, we must follow this 5steps
a) make the partition offline. i.e unmount the drive
b) clean/scan the drive using e2fsck
c) format the drive using resize2fs
d) reduce the logical volume
e) make the partition online i.e mount the drive
Note: During reducing, the data present in the reducing part will be lost
Lets Start with integration with python.
check out the video of automation of lvm with hadoop
GitHub: https://github.com/Anuddeeph/Task7.1-Automation-of-Hadoop-Datanode-Using-LVM-.git