How We Can Contribute Limited Storage as Slave Node in Distributed Cluster (Hadoop Cluster)?
--
Sometimes in the Hadoop cluster ,it may required that we do not want contribute the whole storage as Slave (Data Node).To solve this challenge i am providing one of the way here using partition concept.
So Let’s Start:
I have already created the hadoop cluster on the top of AWS cloud having 1 master (Name Node) and 1 slave node (Data Node).
For storage i have Created the EBS volume of 8 GB and attached it to Data Node (Slave).
Previously i have contributed whole storage (10 GB) of data node as we can see below :
Suppose now i have the requirement to contribute the 5GB of EBS volume i have created (i.e of 8GB in size).
For this i have to create partition 5GB ,to create partition first we have to go inside the volume with help of following command:
#fdisk <drive> (see in the image)
now with help internal command (See below ) of the drive i have created the primary partition of 5 GB .
after creating partition ,Now i have created the new directory /dnode then i have format the partion /dev/xvdf1 by command
# mkfs.ext4 /dev/xvdf1
atlast i have mounted the /dev/xvdf1 with /dnode
As we can see in the image:
I have successfully created the partition of 5GB .
Now move to another part i.e we have to made some changes in the hdfs configuration file.
Now Go to /etc/hadoop and then edit hdfs-site.xml
Previous:
After Edit:
That’s it we have done all the steps.
After Starting the hadoop :
now let’s see what happened
We can clearly see that i have successfully contribute the storage of 5GB of my EBS Volume.
For any query ping to me on linkedin
Source:
