Skip to content

Instantly share code, notes, and snippets.

@nkrumm

nkrumm/aspera_to_s3.sh

Created December 23, 2013 07:14
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nkrumm/8092870 to your computer and use it in GitHub Desktop.
Save nkrumm/8092870 to your computer and use it in GitHub Desktop.
Download ZIP
DIY Direct-to-S3 Aspera solution. This snipped monitors a directory for new files (uploaded via aspera, for example) and then starts a background s3cmd job to upload those files Using the --partial-file-suffix=.part for the ascp command ensures that inotifywait only gets trigger when the file is finished uploading.
Raw
aspera_to_s3.sh
inotifywait -m -e moved_to $BASE | while read line
do
F=$(echo $line | cut -f3 -d" ")
echo "Now uploading $F"
s3cmd --force --no-progress put $F s3://mybucket/ &
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment