Copy between S3 buckets w/ different accounts

S3 buckets copy.md

This is a mix between two sources:

• http://www.alfielapeter.com/posts/8-transferring-s3-bucket-contents-between-accounts-with-s3cmd
• and http://blog.techopsguru.com/2011/12/s3-bucket-copying-with-multiple-accounts.html

basically the first resource is great but didn't work for me: I had to remove the trailing "/*" in the resource string to make it work. I also noticed that setting the policy on the source bucket was sufficient. In the end these are the exact steps I followed to copy data between two buckets on two accounts

Basically the idea there is:

• we allowe the destination account to read the source bucket (in the console for the source account)
• we log as the destination and start the copy

Step 1 grab the user name for the destination account

log into AWS with the destination account and go to "My Account" https://portal.aws.amazon.com/gp/aws/manageYourAccount The account number is on the top right below the search bar (under "Welcome XXX") and is like 1234-1234-1234 (12 digits)

For the rest I also assume you have and API key/password, if not:

• go to the the console https://console.aws.amazon.com

• click on your name on the top right > Security Credentials

• Expand "Access Keys" and click on "Create New Access Key" You then obtain a file that looks like that:

  AWSAccessKeyId=AAAAAAAAAA AWSSecretKey=abababababababababababaabbabab

The first value (AAAAAAAAAA) is the API key, the second (abababababababababababaabbabab) is the password.

Step 2 create the policy for the source bucket

log into AWS with the source account and go to the AWS console for S3

select your bucket > Properties (on the right) > Permissions > Edit bucket policy You then see a dialog named "Bucket Policy Editor"

on the bottom left of the dialog select "AWS policy generator". It will open a new page with a form, set the following values:

• Select Type of Policy: S3 Bucket Policy
• Effect: Allow
• Principal: arn:aws:iam::123412341234:root (123412341234 is the destination account number without the dashes)
• AWS Service: Amazon S3
• Actions: click "All Actions"
• Amazon Resource Name: arn:aws:s3:::source-bucket (replace "source-bucket" with your source bucket name)

The click "Add Statement" and then "Generate Policy" You then see a dialog with contents similar to:

{
  "Id": "Policy1383062241257",
  "Statement": [
    {
      "Sid": "Stmt1383062239775",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::source-bucket",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123412341234:root"
        ]
      }
    }
  ]
}

cut and paste the policy in the dialog of the previous page (the "Bucket Policy Editor") and click "Save"

Step 3 copy using s3cmd

Install s3cmd, on the Mac:

brew install s3cmd

then configure your credentials for the destination account:

s3cmd --configure

It will ask for your API key and corresponding password, then a password to encode your credentials. Andswer yes (y) to test the connection and save the configuration.

now you can copy:

s3cmd sync --skip-existing --recursive s3://source-bucket s3://destination-bucket