Skip to content

Instantly share code, notes, and snippets.

@t3rmin4t0r
Created June 10, 2013 16:58
Show Gist options
  • Save t3rmin4t0r/5750396 to your computer and use it in GitHub Desktop.
Save t3rmin4t0r/5750396 to your computer and use it in GitHub Desktop.
hadoop pull-logs script
#!/bin/bash
for JOBID in $*; do
ATTEMPTS=$(mapred job -list-attempt-ids $JOBID MAP completed; mapred job -list-attempt-ids $JOBID REDUCE completed)
mkdir -p $JOBID
echo $ATTEMPTS | xargs -P 32 -n 1 bash -c "mapred job -logs $JOBID \$0 > $JOBID/\$0.log" 2> /dev/null
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment