Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Empty an s3 bucket of all object versions and delete markers in batches
#!/usr/bin/env bash
set -eEo pipefail
shopt -s inherit_errexit >/dev/null 2>&1 || true
if [[ ! "$#" -eq 2 || "$1" != --bucket ]]; then
echo -e "USAGE: $(basename "$0") --bucket <bucket>"
exit 2
fi
# $@ := bucket_name
empty_bucket() {
local -r bucket="${1:?}"
for object_type in Versions DeleteMarkers; do
local opt=() next_token=""
while [[ "$next_token" != null ]]; do
page="$(aws s3api list-object-versions --bucket "$bucket" --output json --max-items 400 "${opt[@]}" \
--query="[{Objects: ${object_type}[].{Key:Key, VersionId:VersionId}}, NextToken]")"
objects="$(jq -r '.[0]' <<<"$page")"
next_token="$(jq -r '.[1]' <<<"$page")"
case "$(jq -r .Objects <<<"$objects")" in
'[]'|null) break;;
*) opt=(--starting-token "$next_token")
aws s3api delete-objects --bucket "$bucket" --delete "$objects";;
esac
done
done
}
empty_bucket "${2#s3://}"
@jobwat

This comment has been minimized.

Copy link

@jobwat jobwat commented Dec 1, 2019

Thanks! Worked like a charm 👍

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jan 13, 2020

hi , i am getting this error
line 25: /usr/local/bin/aws: Argument list too long
any suggestion ?

@wknapik

This comment has been minimized.

Copy link
Owner Author

@wknapik wknapik commented Jan 19, 2020

Hi @michaelm88. You can change the 1000 on line 17 to a lower value to sort it out. I'm curious what system you've seen this on. Check out https://www.in-ulm.de/~mascheck/various/argmax/ to find out more.

@zishanj

This comment has been minimized.

Copy link

@zishanj zishanj commented May 25, 2020

Any possibility to delete files older than specific number of days only? Like delete all versions older than 7 days. It will be useful to control the storage space and maintain the backup based on weekly basis.

@bitlifter

This comment has been minimized.

Copy link

@bitlifter bitlifter commented Jul 29, 2020

Super handy, thx. after some limited testing, had to limit max-items to 400 but works great!

@wknapik

This comment has been minimized.

Copy link
Owner Author

@wknapik wknapik commented Jan 4, 2021

@zishanj I only ever needed to delete them all, to be able to delete the bucket. But you could probably achieve that just by modifying the page query on line 18.

@bitlifter thanks for the hint, I lowered the number in the script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.