Skip to content

Instantly share code, notes, and snippets.

@manuzhang
Created December 12, 2013 03:45
Show Gist options
  • Save manuzhang/7922886 to your computer and use it in GitHub Desktop.
Save manuzhang/7922886 to your computer and use it in GitHub Desktop.
extract data from log using awk and tr
#!/bin/sh -
JOB_DIR=$1
NUM=$2
UNCOMP=0
for TASK in `ls $JOB_DIR | grep 'attempt.*_m_.*'`
do
ucp=`awk '/.*[\[]MapOutputCollector::mid_spill[\]].*/ { print $28 }' $JOB_DIR/$TASK/stderr | tr -cd [:digit:]`
let "UNCOMP += ucp"
done
let " OUTPUT = $UNCOMP / $NUM / (1024 * 1024) "
echo $OUTPUT MB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment