Bash Script (infinite loop) to Upload Files to AWS S3 Glacier

process_queue.sh

#!/bin/bash

QUEUE_DIR="./queue"
COMPLETED_DIR="./completed"
S3_BUCKET_NAME="<YOUR_BUCKET_NAME_HERE>"
OLDEST_FILENAME=""

echo "Running..."
while :; do
	echo "Finding oldest file..."
	OLDEST_FILENAME="$(find $QUEUE_DIR -type f -printf '%T+ %p\n' | sort | head -n 1 | grep -o " .*")"

	if [[ -n "$OLDEST_FILENAME" ]] ; then
		echo "Found $OLDEST_FILENAME"
		echo "Uploading $OLDEST_FILENAME to S3..."
		aws s3 cp --storage-class=GLACIER --force-glacier-transfer --only-show-errors $OLDEST_FILENAME s3://$S3_BUCKET_NAME
		echo "Done Uploading. Moving to $COMPLETED_DIR..."
		mv $OLDEST_FILENAME $COMPLETED_DIR
	else
		sleep 1s
	fi
	sleep 30s
done

TODO:

• Set directories from command line args
• Read response from aws-cli to make sure it succeeds
• Implement a wait on large files (e.g. tar, zip, mp4, etc.) as they might still be writing to QUEUE_DIR.

This script now has a home at https://github.com/jeffrade/proxy-server