This is a mix between two sources:
- http://www.alfielapeter.com/posts/8-transferring-s3-bucket-contents-between-accounts-with-s3cmd
- and http://blog.techopsguru.com/2011/12/s3-bucket-copying-with-multiple-accounts.html
basically the first resource is great but didn't work for me: I had to remove the trailing "/*" in the resource string to make it work. I also noticed that setting the policy on the source bucket was sufficient. In the end these are the exact steps I followed to copy data between two buckets on two accounts
Basically the idea there is:
- we allowe the destination account to read the source bucket (in the console for the source account)
- we log as the destination and start the copy
log into AWS with the destination account and go to "My Account" https://portal.aws.amazon.com/gp/aws/manageYourAccount The account number is on the top right below the search bar (under "Welcome XXX") and is like 1234-1234-1234 (12 digits)
For the rest I also assume you have and API key/password, if not:
-
go to the the console https://console.aws.amazon.com
-
click on your name on the top right > Security Credentials
-
Expand "Access Keys" and click on "Create New Access Key" You then obtain a file that looks like that:
AWSAccessKeyId=AAAAAAAAAA AWSSecretKey=abababababababababababaabbabab
The first value (AAAAAAAAAA) is the API key, the second (abababababababababababaabbabab) is the password.
log into AWS with the source account and go to the AWS console for S3
select your bucket > Properties (on the right) > Permissions > Edit bucket policy You then see a dialog named "Bucket Policy Editor"
on the bottom left of the dialog select "AWS policy generator". It will open a new page with a form, set the following values:
- Select Type of Policy: S3 Bucket Policy
- Effect: Allow
- Principal: arn:aws:iam::123412341234:root (123412341234 is the destination account number without the dashes)
- AWS Service: Amazon S3
- Actions: click "All Actions"
- Amazon Resource Name: arn:aws:s3:::source-bucket (replace "source-bucket" with your source bucket name)
The click "Add Statement" and then "Generate Policy" You then see a dialog with contents similar to:
{
"Id": "Policy1383062241257",
"Statement": [
{
"Sid": "Stmt1383062239775",
"Action": "s3:*",
"Effect": "Allow",
"Resource": "arn:aws:s3:::source-bucket",
"Principal": {
"AWS": [
"arn:aws:iam::123412341234:root"
]
}
}
]
}
cut and paste the policy in the dialog of the previous page (the "Bucket Policy Editor") and click "Save"
Install s3cmd, on the Mac:
brew install s3cmd
then configure your credentials for the destination account:
s3cmd --configure
It will ask for your API key and corresponding password, then a password to encode your credentials. Andswer yes (y) to test the connection and save the configuration.
now you can copy:
s3cmd sync --skip-existing --recursive s3://source-bucket s3://destination-bucket
Thank you very much for this. Saved me a ton of time today.