Last active
August 31, 2017 07:41
-
-
Save bepcyc/5092264117dc19830f0eedface51790b to your computer and use it in GitHub Desktop.
Preparing disks for hadoop HDFS
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# I use these commands with pdsh assuming my worker nodes look the same | |
# I also assume that my hard disks are /dev/sdb - /dev/sdj | |
for d in {b..j}; | |
do | |
# convert letters b..j to numbers 0..8 | |
dnum=$(python -c "print(ord('${d}')-98)") | |
disk="/dev/sd${d}" | |
umount ${disk}1 | |
mount_point="/grid/${dnum}" | |
disk_label="HDFS${dnum}" | |
# apply GPT partition table | |
sudo parted -s ${disk} mklabel gpt | |
# on each disk create 1 partition with 1MiB alignment | |
sudo parted -s -a optimal ${disk} mkpart ${disk_label} '1024KiB' '100%' | |
# some tuning for hdfs and big files | |
sudo mkfs.ext4 -F -T largefile -m 0 -O dir_index,extent,sparse_super -L ${disk_label} ${disk}1 | |
# force fsck | |
sudo e2fsck -f ${disk}1 | |
# create a mount point | |
sudo mkdir -p ${mount_point} | |
# add new entry to fstab | |
echo -e "LABEL=${disk_label}\t${mount_point}\text4\tdefaults,data=writeback,noatime\t0\t0" | sudo tee -a /etc/fstab | |
# mount a new entry | |
sudo mount -a | |
# remove all the garbage from fsck and make it clean | |
sudo rm -rf ${mount_point}/* | |
done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment