Bidirectional sync between two s3 buckets

bidirectional_sync_s3.md

1. Copy new files in [ENV]-wowzer-iphone to [ENV]-wowza-repository

1.1 Configure the SNS topic

1.1.1 Create a new SNS topic

Create a new SNS topic with name: [ENV]-wowzer-iphone-fanout

1.1.2 Update SNS topic policy

Go to Other topic actions > Edit topic policy > Advanced View and replace the contents of the default policy with the following:

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:eu-west-1:956149323110:[ENV]-wowzer-iphone-fanout",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::[ENV]-wowzer-iphone"
        }
      }
    }
  ]
}

1.2 Configure the S3 bucket

1.2.1 Configure a new event

Create a new Event CopyNewFiles selecting only the following options: Post, Put and Multipart Upload.

Select the SNS topic with name [ENV]-wowzer-iphone-fanout and save.

1.3 Configure the Lambda function

1.3.1 Create the new Lambda function

Enter the following function name: [ENV]-wowza-repository

Select Python 2.7 as Runtime and enter the following code:

import urllib
import boto3
import ast
import json
print('Loading function')

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sns_message = ast.literal_eval(event['Records'][0]['Sns']['Message'])
    target_bucket = context.function_name
    source_bucket = str(sns_message['Records'][0]['s3']['bucket']['name'])
    key = str(urllib.unquote_plus(sns_message['Records'][0]['s3']['object']['key']).decode('utf8'))
    copy_source = {'Bucket':source_bucket, 'Key':key}
    print "Copying %s from bucket %s to bucket %s ..." % (key, source_bucket, target_bucket)
    s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)

Leave the default handler and create a custom role with name lambda_basic_execution_[ENV]_wowza_repository and the following IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::[ENV]-wowzer-iphone/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::[ENV]-wowza-repository/*"
            ]
        }
    ]
}

Change the function's timeout to 5 minutes.

1.3.2 Create the SNS topic subscription

From the SNS console, create a new subscription for the topic [ENV]-wowzer-iphone-fanout and the Lambda function [ENV]-wowza-repository.

1.4 Test one direction sync-ing

Upload a file in [ENV]-wowzer-iphone and make sure it gets properly copied to [ENV]-wowza-repository.

2. Copy new files in [ENV]-wowza-repository to [ENV]-wowzer-iphone

2.1 Configure the SNS topic

2.1.1 Create a new SNS topic

Create a new SNS topic with name: [ENV]-wowza-repository-fanout

2.1.2 Update SNS topic policy

Go to Other topic actions > Edit topic policy > Advanced View and replace the contents of the default policy with the following:

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:eu-west-1:956149323110:[ENV]-wowza-repository-fanout",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::[ENV]-wowza-repository"
        }
      }
    }
  ]
}

2.2 Configure the S3 bucket

2.2.1 Configure a new event

Create a new Event CopyNewFiles selecting only the following options: Post, Put and Multipart Upload.

Select the SNS topic with name [ENV]-wowza-repository-fanout and save.

2.3 Configure the Lambda function

2.3.1 Create the new Lambda function

Enter the following function name: [ENV]-wowzer-iphone

Select Python 2.7 as Runtime and enter the following code:

import urllib
import boto3
import ast
import json
print('Loading function')

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sns_message = ast.literal_eval(event['Records'][0]['Sns']['Message'])
    target_bucket = context.function_name
    source_bucket = str(sns_message['Records'][0]['s3']['bucket']['name'])
    key = str(urllib.unquote_plus(sns_message['Records'][0]['s3']['object']['key']).decode('utf8'))
    copy_source = {'Bucket':source_bucket, 'Key':key}
    print "Copying %s from bucket %s to bucket %s ..." % (key, source_bucket, target_bucket)
    s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)

Leave the default handler and create a custom role with name lambda_basic_execution_[ENV]_wowzer_iphone and the following IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::[ENV]-wowza-repository/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::[ENV]-wowzer-iphone/*"
            ]
        }
    ]
}

Change the function's timeout to 5 minutes.

2.3.2 Create the SNS topic subscription

From the SNS console, create a new subscription for the topic [ENV]-wowza-repository-fanout and the Lambda function [ENV]-wowzer-iphone.

2.4 Test two direction sync-ing

Upload a file in [ENV]-wowzer-iphone and make sure it gets properly copied to [ENV]-wowza-repository. Upload a file in [ENV]-wowza-repository and make sure it gets properly copied to [ENV]-wowzer-iphone.

3. Copy existing files in [ENV]-wowzer-iphone to [ENV]-wowza-repository

3.1 Copy all existing files to new repository

Use AWS S3 console to copy all existing files in [ENV]-wowzer-iphone to [ENV]-wowza-repository.