Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 14 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save pecigonzalo/3f89a4b29b6fae933000ca03720e15c5 to your computer and use it in GitHub Desktop.
Save pecigonzalo/3f89a4b29b6fae933000ca03720e15c5 to your computer and use it in GitHub Desktop.
Delete all versions (except latest) of all files in s3 versioned bucket using AWS CLI and jq.
#!/bin/bash
bucket=$1
set -e
echo "Removing all versions from $bucket"
versions=`aws s3api list-object-versions --bucket $bucket |jq '.Versions | .[] | select(.IsLatest | not)'`
markers=`aws s3api list-object-versions --bucket $bucket |jq '.DeleteMarkers'`
echo "removing files"
for version in $(echo "${versions}" | jq -r '@base64'); do
version=$(echo ${version} | base64 --decode)
key=`echo $version | jq -r .Key`
versionId=`echo $version | jq -r .VersionId `
cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
echo $cmd
$cmd
done
echo "removing delete markers"
for marker in $(echo "${markers}" | jq -r '.[] | @base64'); do
marker=$(echo ${marker} | base64 --decode)
key=`echo $marker | jq -r .Key`
versionId=`echo $marker | jq -r .VersionId `
cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
echo $cmd
$cmd
done
@Tarvinder91
Copy link

Tarvinder91 commented Nov 17, 2020

@nashjain your solution is more efficient as it deletes 1000 objects at a time and its blazing faster than deleting 1 object at a time. I had millions of objects so listing all the objects made the list query getting time out so we needed to limit the results.

Also very IMP- if you delete the delete marker and object has previous versions still existing, the object will start appearing in bucket as if its not deleted. So very imp that delete markers are only deleted when previous version objects are fully deleted.

My comprehensive script to delete the previous versions of object except the latest one is here.

Another caveat, if u have more than half million objects under a prefix, use this command to limit the results :
aws s3api list-object-versions --prefix $prefix --bucket $bucket --max-items 300000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment