Skip to content

Instantly share code, notes, and snippets.

View zachliu's full-sized avatar

Zach Liu zachliu

  • New York
View GitHub Profile
@halilduygulu
halilduygulu / s3_bulk_delete_cli.sh
Last active April 24, 2023 17:57
AWS S3 CLI command to filter and delete objects by file size from bucket, 1000 by 1000 using s3api delete-objects command.
### S3 Bulk Delete by File Size ###
aws s3 ls s3://YOUR_BUCKET_NAME/content/article/data/ --profile YOUR_AWS_PROFILE --recursive | awk -F ' ' '{print $3,$4}' | awk -F ' ' '$1 < 100 {print $2}' | xargs -IP echo '{"Key": "P"}' > delete.txt
#Because bulk delete limit is 1000 per api call.
split -l 1000 delete.txt
#Append json start and end parts to complete a bulk delete request in every file.
for file in x*; do
echo '{"Objects": [' >> delete"$file" && paste -d, -s "$file" >> delete"$file" &&
echo '],"Quiet": true }' >> delete"$file"