Skip to content

Instantly share code, notes, and snippets.

@t3rmin4t0r

t3rmin4t0r/pull-logs.sh

Created June 10, 2013 16:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save t3rmin4t0r/5750396 to your computer and use it in GitHub Desktop.
Save t3rmin4t0r/5750396 to your computer and use it in GitHub Desktop.
Download ZIP
hadoop pull-logs script
Raw
pull-logs.sh
#!/bin/bash
for JOBID in $*; do
ATTEMPTS=$(mapred job -list-attempt-ids $JOBID MAP completed; mapred job -list-attempt-ids $JOBID REDUCE completed)
mkdir -p $JOBID
echo $ATTEMPTS | xargs -P 32 -n 1 bash -c "mapred job -logs $JOBID \$0 > $JOBID/\$0.log" 2> /dev/null
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment