Skip to content

Instantly share code, notes, and snippets.

@olen2006
Forked from shapeshed/aws-cf-logs
Created May 12, 2019 01:40
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save olen2006/d68292f86fbedc4b25a1ff88c870809d to your computer and use it in GitHub Desktop.
Fetch AWS Cloudfront Logs, decompress, combine into a single file and remove comments
#!/usr/bin/env bash
BUCKET=$1
CWD=$(pwd)
if [[ -n $1 ]]; then
aws s3 sync s3://$BUCKET/cf-logs .
cat *.gz > combined.log.gz
find $CWD ! -name 'combined.log.gz' -name '*.gz' -type f -exec rm -f {} +
gzip -d combined.log.gz
sed -i '/^#/ d' combined.log
exit 0
else
echo "Error: no bucket name provided"
exit 1
fi
@olen2006
Copy link
Author

https://shapeshed.com/aws-cloudfront-log/
The script does the following:

Reads the bucket name as the first argument
Synchronises the current working directory with the specified S3 bucket
Combines the gzipped log files into a single file
Removes all files other than the combined file
Decompresses the file
Removes comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment