Skip to content

Instantly share code, notes, and snippets.

@kimsterv
Created June 22, 2011 18:23
Show Gist options
  • Save kimsterv/1040741 to your computer and use it in GitHub Desktop.
Save kimsterv/1040741 to your computer and use it in GitHub Desktop.
Clean bad gzips
#!/bin/bash
for i in `hadoop fs -lsr /flume/incoming/2011/04 | awk '{print $8}'`; do
echo $i; hadoop fs -cat $i | gzip -t
if [ $? -ne 0 ] && [ -n "`echo \"$i\" | grep 'gz$'`" ]; then
`echo hadoop fs -rmr -skipTrash $i`
fi
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment