Skip to content

Instantly share code, notes, and snippets.

@TechNickAI
Created December 4, 2012 19:56
Show Gist options
  • Save TechNickAI/4208023 to your computer and use it in GitHub Desktop.
Save TechNickAI/4208023 to your computer and use it in GitHub Desktop.
bash script for pulling files from all servers
#!/bin/bash
### Pulls files for a specific .gif into the local directory, uncompresses them,
### and creates a all.$gif.log file with all of the entries
if [ "$1" == "" ]; then
day=`date +%Y-%m-%d`
else
day=$1
fi
gif=sentry.gif
daydir=/mnt/tmp/$gif/$day
echo Pulling files for $day to $daydir
mkdir -p $daydir
alllog=$daydir/all.$gif.log
rm -f $alllog
echo "Querying cloudkick for a list of beacon servers"
for box in `cloudkick list --query 'node:beacon-* and penv:production' | grep -v IP | awk '{print $1}' | sort | uniq`; do
mkdir -p $daydir/$box/
rm -f $daydir/$box/*.log
echo Transferring from $box
rsync -ae ssh $box:/data/var/log/apache2/$gif-$day* $daydir/$box
echo Decompressing
lzop -d $daydir/$box/*.lzo
echo Appending to $alllog
cat $daydir/$box/*.log >> $alllog
done
echo Done. All log files combined in $alllog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment