Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Fetch AWS Cloudfront Logs, decompress, combine into a single file and remove comments
#!/usr/bin/env bash
BUCKET=$1
CWD=$(pwd)
if [[ -n $1 ]]; then
aws s3 sync s3://$BUCKET/cf-logs .
cat *.gz > combined.log.gz
find $CWD ! -name 'combined.log.gz' -name '*.gz' -type f -exec rm -f {} +
gzip -d combined.log.gz
sed -i '/^#/ d' combined.log
exit 0
else
echo "Error: no bucket name provided"
exit 1
fi
@olen2006

This comment has been minimized.

Copy link
Owner Author

olen2006 commented May 12, 2019

https://shapeshed.com/aws-cloudfront-log/
The script does the following:

Reads the bucket name as the first argument
Synchronises the current working directory with the specified S3 bucket
Combines the gzipped log files into a single file
Removes all files other than the combined file
Decompresses the file
Removes comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.