Skip to content

Instantly share code, notes, and snippets.

@victorfsf
Last active June 12, 2017 14:44
Show Gist options
  • Save victorfsf/cb965515e97353017279 to your computer and use it in GitHub Desktop.
Save victorfsf/cb965515e97353017279 to your computer and use it in GitHub Desktop.
Downloading and uploading from/to a S3 Bucket using AWS CLI

Installing & Configuring

$ sudo pip install awscli (or: sudo apt-get install awscli)
$ aws configure

You'll need to fill the following settings:

AWS Access Key ID [None]: 
AWS Secret Access Key [None]: 
Default region name [None]: 
Default output format [None]: 

Downloading from S3

$ aws s3 sync s3://bucket_name .

Uploading to S3

$ aws s3 sync files/to/upload/path s3://bucket_name

Copying from Bucket to Bucket

You'll need to set the following policies for both Buckets:

Source Bucket policy statements:

{
	"Sid": "Stmt1357935647218",
	"Effect": "Allow",
	"Principal": {
		"AWS": "arn:aws:iam::XXXXXXXXXXXX:user"
	},
	"Action": "s3:ListBucket",
	"Resource": "arn:aws:s3:::SourceBucket"
},
{
	"Sid": "Stmt1357935676138",
	"Effect": "Allow",
	"Principal": {
		"AWS": "arn:aws:iam::XXXXXXXXXXXX:user"
	},
	"Action": "s3:GetObject",
	"Resource": "arn:aws:s3:::SourceBucket/*"
},

Destination Bucket policy statements:

{
	"Sid": "Stmt1357935647218",
	"Effect": "Allow",
	"Principal": {
		"AWS": "arn:aws:iam::XXXXXXXXXXXX:user"
	},
	"Action": "s3:ListBucket",
	"Resource": "arn:aws:s3:::DestinationBucket"
},
{
	"Sid": "Stmt1357935676138",
	"Effect": "Allow",
	"Principal": {
		"AWS": "arn:aws:iam::XXXXXXXXXXXX:user"
  },
	"Action": "s3:PutObject",
	"Resource": "arn:aws:s3:::DestinationBucket/*"
}

Once you set these policies, just run:

$ aws s3 cp s3://SourceBucket/ s3://DestinationBucket/ --recursive
  • The --recursive argument will download everything inside the specified folder. Without this argument, you'd have to download each file, one by one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment