Skip to content

Instantly share code, notes, and snippets.

@Gershon-A
Last active January 31, 2024 12:04
Show Gist options
  • Save Gershon-A/5fadb55289c0aa6f265d155bf46a6758 to your computer and use it in GitHub Desktop.
Save Gershon-A/5fadb55289c0aa6f265d155bf46a6758 to your computer and use it in GitHub Desktop.
This script lists all objects in a specified AWS S3 bucket that were moved to the GLACIER storage class within a specified number of days ago.
#!/bin/bash
# This script lists all objects in a specified AWS S3 bucket that were moved to the GLACIER storage class within a specified number of days ago.
# Usage: ./show_obgects_moved_to_glacier.sh your-bucket-name aws-profile days_ago
# Get the arguments
bucket="$1"
profile="$2"
days_ago="$3"
# Set the page size for pagination
page_size=100
# Initialize a counter for the number of objects
count=0
# Initialize a counter for the number of objects processed
processed=0
# Calculate the date days_ago days ago in Unix timestamp in UTC
date_days_ago=$(date -u -d "$days_ago days ago" +%s)
# List all objects in your bucket
while read -r object; do
key=$(echo "$object" | jq -r '.[0]')
last_modified=$(echo "$object" | jq -r '.[1]')
storage_class="GLACIER"
# Convert the LastModified date to a Unix timestamp
last_modified_unix=$(date -d"$last_modified" -u +%s)
# Debug: Print the key, last modified date, and storage class of the object for debugging
# echo "Key: $key, Last Modified: $last_modified, Storage Class: $storage_class"
# Check if the storage class is GLACIER and the LastModified date is greater than or equal to date_days_ago
if [ "$last_modified_unix" -ge "$date_days_ago" ]; then
# Print the key of the object
echo "Key: $key"
# Increment the counter
((count++))
fi
# Increment the processed counter
((processed++))
# Print the progress every 100 objects
if (( processed % 100 == 0 )); then
echo "Processed $processed objects so far"
fi
done < <(aws s3api list-objects-v2 --bucket $bucket --query 'Contents[?StorageClass==`GLACIER`].[Key, LastModified]' --output json --profile $profile --page-size $page_size | jq -c '.[]')
# Print the summary
echo "Total number of objects moved to GLACIER in the last $days_ago days: $count"
@Gershon-A
Copy link
Author

Gershon-A commented Jan 31, 2024

Script: show_objects_moved_to_glacier.sh

This script lists all objects in a specified AWS S3 bucket that were moved to the GLACIER storage class within a specified number of days ago.

Usage

./show_objects_moved_to_glacier.sh <your-bucket-name> <aws-profile> <days_ago>

Parameters

  • <your-bucket-name>: The name of the AWS S3 bucket to check.
  • <aws-profile>: The AWS profile to use for AWS CLI commands.
  • <days_ago>: The number of days ago to check for objects moved to GLACIER.

How it works

  1. The script first initializes some variables based on the command line arguments, including the bucket name, AWS profile, and number of days ago to check.
  2. It calculates the date <days_ago> days ago in Unix timestamp in UTC.
  3. The script then lists all objects in the specified bucket that have a storage class of GLACIER.
  4. For each object, it extracts the key and the LastModified date.
  5. The LastModified date is converted to a Unix timestamp.
  6. The script then checks if the LastModified date is greater than or equal to the date <days_ago> days ago. If it is, the script increments a counter.
  7. Finally, the script prints the total number of objects that were moved to GLACIER within the specified number of days ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment