Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Fetch AWS Cloudfront Logs, decompress, combine into a single file and remove comments
#!/usr/bin/env bash
if [[ -n $1 ]]; then
aws s3 sync s3://$BUCKET/cf-logs .
cat *.gz > combined.log.gz
find $CWD ! -name 'combined.log.gz' -name '*.gz' -type f -exec rm -f {} +
gzip -d combined.log.gz
sed -i '/^#/ d' combined.log
exit 0
echo "Error: no bucket name provided"
exit 1
Copy link

olen2006 commented May 12, 2019
The script does the following:

Reads the bucket name as the first argument
Synchronises the current working directory with the specified S3 bucket
Combines the gzipped log files into a single file
Removes all files other than the combined file
Decompresses the file
Removes comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment