Skip to content

Instantly share code, notes, and snippets.

@nashjain
Last active July 25, 2022 21:09
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save nashjain/6119aecd5e8919d0818773a118d05ed6 to your computer and use it in GitHub Desktop.
Save nashjain/6119aecd5e8919d0818773a118d05ed6 to your computer and use it in GitHub Desktop.
Let's assume you backup a file every hour on AWS S3 and you want to clean up versions that are older than a week. However for the older versions, you want to leave one version per day just in case if you need them. Following script does that for you.
#!/bin/bash
deleteBefore=`date --date="1 week ago" +%F`
bucket=$1
fileToDelete=$2
fileName='aws_delete.json'
rm $fileName
echo "Removing all versions of $fileToDelete from $bucket"
versionsToDelete=`aws s3api list-object-versions --bucket "$bucket" --prefix "$fileToDelete" --query "Versions[?(LastModified<'$deleteBefore' && (contains(LastModified, 'T0') || contains(LastModified, 'T1') || contains(LastModified, 'T20') || contains(LastModified, 'T21') || contains(LastModified, 'T22')))].{Key: Key, VersionId: VersionId}"`
cat << EOF > $fileName
{"Objects":$versionsToDelete, "Quiet":true}
EOF
aws s3api delete-objects --bucket "$bucket" --delete file://$fileName
# s3api delete-objects can handle upto 1000 records
echo "Delete successsful"
# We leave the aws_delete.json file, in case you want to later see what happened.
@tarvinder1
Copy link

If the obj list is greater than 1000, the operation times out. Better to handle the objects in loop of some kind

@Tarvinder91
Copy link

For deleting all objects with the above command, use this snippet to loop through all object list 1000 objects at a time:
#Get old version Objects
echo "Old version objects under this prefix:"
cat "all-objects-$bucket.json" | jq '.[] | select(.IsLatest | not)' | jq -s '.' > "old-objects-$bucket.json"
no_of_obj=$(cat old-objects-$bucket.json | jq 'length')
i=0
while [ $i -lt $no_of_obj ]
do
next=$((i+999))
oldversions=$(cat "old-objects-$bucket.json" | jq '.[] | {Key,VersionId}' | jq -s '.' | jq .[$i:$next])
cat << EOF > deleted-files-start-index-$i.json
{"Objects":$oldversions, "Quiet":true}
EOF
echo "Deleting records from $i - $next"
aws s3api delete-objects --bucket "$bucket" --delete file://deleted-files-start-index-$i.json
let i=i+1000
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment