Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@ryanjdillon

ryanjdillon/s3-rsync.sh

Last active July 2, 2022 12:23
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ryanjdillon/46fdca9b131cb45d2a90bcff5608f998 to your computer and use it in GitHub Desktop.
Save ryanjdillon/46fdca9b131cb45d2a90bcff5608f998 to your computer and use it in GitHub Desktop.
Download ZIP
Sync data to S3 with `aws s3 sync`, an AWS `rsync` emulator
Raw
s3-rsync.sh
# Install the aws cli
source your-venv/bin/activate
pip install awscli
# Setup your credentials
aws configure
# Check out what is going to get transfered first
aws s3 sync ./path/to/data s3://bucket-name --exclude "*" --include "*.nc" --dryrun
# Remove --dryrun to sync for realsies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment