Skip to content

Instantly share code, notes, and snippets.

@vitalibertas
Created August 13, 2017 03:51
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save vitalibertas/d17bba1219ed22e42f6b018608b85b96 to your computer and use it in GitHub Desktop.
Check HDFS for a specific file a certain amount of times before it errors out so you don't execute code that has a dependency.
CHECK_HDFS="/some/path/to/file"
function hdfsCheck {
RETRY=0
while [ $RETRY -lt 9 ];
do
COUNT=$(hdfs dfs -ls "${CHECK_HDFS}" | wc -l) 2> stderr.txt
if [ $COUNT -lt 1 ]; then
echo "Try $RETRY. Replication hasn't happened yet! $(date +"%Y-%m-%d %H:%M")"
sleep 20m
RETRY=$[ $RETRY + 1 ]
else
echo "Replication occured."
break
fi
done
if [ $RETRY -lt 9 ]; then
return 0
else
return 1
fi
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment