Skip to content

Instantly share code, notes, and snippets.

@rgn
Created February 1, 2019 18:56
Show Gist options
  • Save rgn/3d7b0bd07d99833c53aa248acb039fc9 to your computer and use it in GitHub Desktop.
Save rgn/3d7b0bd07d99833c53aa248acb039fc9 to your computer and use it in GitHub Desktop.
Correct replication factor for all underreplicated blocks in HDFS.
!/bin/env bash
if [ -z ${1} ]; then
echo "Replication factor required."
exit 1
fi
TEMP_FILE=$(mktemp)
echo "Writing underreplicated files to $TEMP_FILE"
hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> $TEMP_FILE
for hdfsfile in `cat $TEMP_FILE`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep $1 $hdfsfile; done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment