Skip to content

Instantly share code, notes, and snippets.

@michailw
Last active April 3, 2019 19:33
Show Gist options
  • Save michailw/d51ba6c76ad769ee7e19dd94ed3f8055 to your computer and use it in GitHub Desktop.
Save michailw/d51ba6c76ad769ee7e19dd94ed3f8055 to your computer and use it in GitHub Desktop.
Cleans up S3 bucket using s3cmd command from older backups
#!/bin/bash
BUCKET_NAME=$1
NOW=$(date +"%s")
NOW_YEAR=$(date +"%Y")
PAST=$(( $NOW - (7 * 3600 * 24) ))
PAST_STRING=$(date -d @$PAST +"%Y-%m-%d")
LOG_PATH="/tmp/bucket-cleanup_$(date +"%Y-%m-%d").log"
DATA=$(s3cmd ls s3://${BUCKET_NAME}/ --recursive | awk -v PAST="${PAST_STRING}" '$1 < PAST {print $0}')
while read -r line; do
BACKUP_PATH=$(echo "$line" | awk '{print $4}')
BACKUP_DAY=$(echo "$line" | awk '{print $1}')
BACKUP_YEAR=$(echo "$BACKUP_DAY" | awk -F "-" '{print $1}')
FIRST_DAY_OF_MONTH=$(echo "$BACKUP_DAY" | awk -F "-" '{print $1"-"$2"-20"}')
if [[ $BACKUP_DAY == $FIRST_DAY_OF_MONTH && $BACKUP_YEAR == $NOW_YEAR ]]; then
continue
fi
if [[ $BACKUP_DAY < $PAST_STRING ]]; then
s3cmd rm $BACKUP_PATH >>$LOG_PATH 2>&1
fi
done <<< "$DATA"
s3cmd put $LOG_PATH s3://${BUCKET_NAME}/cleanup-log/bucket-cleanup_$(date +"%Y-%m-%d").log --quiet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment