Skip to content

Instantly share code, notes, and snippets.

@DangerBlack
Last active June 24, 2024 12:23
Show Gist options
  • Save DangerBlack/7d7e7baee05fd1bca01c62718be1047f to your computer and use it in GitHub Desktop.
Save DangerBlack/7d7e7baee05fd1bca01c62718be1047f to your computer and use it in GitHub Desktop.
How to delete the objects in a versioned bucket
#!/usr/bin/bash
# To configure aws cli on Cubbit you can visit
# https://docs.cubbit.io/integrations/aws-cli#installation
#MIT License
#Copyright (c) [2023] [DangerBlack]
#Permission is hereby granted, free of charge, to any person obtaining a copy
#of this software and associated documentation files (the "Software"), to deal
#in the Software without restriction, including without limitation the rights
#to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#copies of the Software, and to permit persons to whom the Software is
#furnished to do so, subject to the following conditions:
#The above copyright notice and this permission notice shall be included in all
#copies or substantial portions of the Software.
#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#SOFTWARE.
#Please configure those variable
profile="cubbit"
bucket="bucket-name"
if ! command -v parallel &> /dev/null; then
echo "GNU parallel is not installed. Please install it using your package manager."
echo "For example, you can use one of the following commands depending on your system:"
echo " sudo apt-get install parallel # Debian/Ubuntu"
echo " sudo yum install parallel # RedHat/CentOS"
exit 1
fi
if ! command -v jq &> /dev/null; then
echo "jq is not installed. Please install it using your package manager."
echo "For example, you can use one of the following commands depending on your system:"
echo " sudo apt-get install jq # Debian/Ubuntu"
echo " sudo yum install jq # RedHat/CentOS"
exit 1
fi
aws --endpoint https://s3.cubbit.eu s3api list-object-versions --bucket "$bucket" --output=json --query='{Objects: [Versions,DeleteMarkers][].{Key:Key,VersionId:VersionId}}' --profile "$profile" > keys_dump.json
file='keys_dump.json'
jq -r '.Objects | _nwise(200) | "aws --endpoint https://s3.cubbit.eu s3api delete-objects --bucket \"$bucket\" --delete ###{\"Objects\": \(.),\"Quiet\": true}### --profile $profile"' '$file' > remove_keys.sh
sed -i "s/###/'/g" remove_keys.sh
chmod +x remove_keys.sh
echo "Start actively deleting keys"
cat remove_keys.sh | parallel -j 8
#./remove_keys.sh
echo "done"
rm remove_keys.sh
rm keys_dump.json
echo "Now bucket is empty"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment