Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@michaelahlers
Last active April 4, 2019 16:27
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save michaelahlers/bb285de5dd3983b847bebe194b096df5 to your computer and use it in GitHub Desktop.
Save michaelahlers/bb285de5dd3983b847bebe194b096df5 to your computer and use it in GitHub Desktop.
Bulk restore-request of objects in S3.
#!/bin/bash
BUCKET_NAME="my-bucket"
KEY_PREFIX="my-key-prefix"
DAYS = "10"
aws s3 ls --recursive "s3://$BUCKET_NAME/$KEY_PREFIX" |\
awk '{$1=$2=$3=""; print $0}' |\
sed 's/^[[:space:]]*//g' |\
tr '\n' '\0' |\
xargs -t -0 -I {} aws s3api restore-object --restore-request Days=$DAYS --bucket "$BUCKET_NAME" --key "{}"
@michaelahlers
Copy link
Author

michaelahlers commented Mar 28, 2019

Iterating over columns with awk is problematic for object keys with reserved characters. A resilient approach is blanking the date, time, and size columns, printing the remainder, then removing any leading whitespace left over. Following that, xargs is not robust against quotes (producing “xargs: unterminated quote” errors). Convert newlines to null, then use -0 to indicate null-delimited lines.

See also:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment