Skip to content

Instantly share code, notes, and snippets.

@LucasRoesler
Last active February 21, 2024 22:59
Show Gist options
  • Save LucasRoesler/9a865f3924df706d3c0b9e7337ca111f to your computer and use it in GitHub Desktop.
Save LucasRoesler/9a865f3924df706d3c0b9e7337ca111f to your computer and use it in GitHub Desktop.
Example cloudformation that sets up an s3 bucket and notifications sent to an sqs queue.

Usage

Create a new user in the console, assign it no permissions, but make sure to download the credentials.

Go to the user and copy the user ARN:

arn:aws:iam::<account-id>:user/of-connector-test

This can also be done via the CLI

aws iam create-user --user-name of-connector-test
aws iam create-access-key --user-name of-connector-test

Now install the connector stack via the UI or via the cli

USER_ARN=$(aws iam get-user --user-name of-connector-test --output=json | jq -r  ".User.Arn")
aws cloudformation deploy \
  --stack-name of-connector-example \
  --template-file cloudformation.yml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides ConnectorAccountArn=$USER_ARN

Message Format

The messages will be the usual S3 notification format, for example

{
    "Records": [
        {
            "eventVersion": "2.1",
            "eventSource": "aws:s3",
            "awsRegion": "eu-central-1",
            "eventTime": "2021-12-22T13:34:59.690Z",
            "eventName": "ObjectCreated:Copy",
            "userIdentity": {
                "principalId": "AWS:AIDA...."
            },
            "requestParameters": {
                "sourceIPAddress": "95.90.240.192"
            },
            "responseElements": {
                "x-amz-request-id": "KHS4JGPWTAN06JJR",
                "x-amz-id-2": "5MWIyqTq0rIhOFdIu/6jBYcOaPj3...."
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "e76e4eb1-911a-43de-89f9-c1f8541b80a9",
                "bucket": {
                    "name": "eu-central-1-of-connector-ex-bucket",
                    "ownerIdentity": {
                        "principalId": "ABSRQAS2T5Q66"
                    },
                    "arn": "arn:aws:s3:::eu-central-1-of-connector-ex-bucket"
                },
                "object": {
                    "key": "upload/cf-designer.png",
                    "size": 84181,
                    "eTag": "21a4958708420644d20ac12e6ccb28fc",
                    "sequencer": "0061C32983A5EC6864"
                }
            }
        }
    ]
}
AWSTemplateFormatVersion: 2010-09-09
Description: OpenFaaS S3 events connector template
Parameters:
ConnectorAccountArn:
Type: String
Description: The ARN of the connector account IAM user
Resources:
# create a queue for our s3 upload events
S3Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Join
- "-"
- - !Ref "AWS::Region"
- !Ref "AWS::StackName"
- queue
# enable long polling by default to reduce the number
# of requests to the SQS API
ReceiveMessageWaitTimeSeconds: 20
# The length of time during which a message will be unavailable
# after a message is delivered from the queue. This blocks other
# components from receiving the same message and gives the initial
# component time to process and delete the message from the queue.
#
# CloudFormation default is 30 seconds.
# Must be from 0 to 43,200 seconds (12 hours)
VisibilityTimeout: 30
# create a bucket to upload files to and send
# upload events to our queue, only if the file
# has the prefix (is in the folder) /upload
S3Bucket:
Type: "AWS::S3::Bucket"
# DeletionPolicy: Retain
# UpdateReplacePolicy: Retain
Properties:
BucketName: !Join
- "-"
- - !Ref "AWS::Region"
- !Ref "AWS::StackName"
- bucket
NotificationConfiguration:
QueueConfigurations:
- Event: s3:ObjectCreated:*
Queue: !GetAtt S3Queue.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: upload/
# allow the bucket to send messages
SQSQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS: !Ref ConnectorAccountArn
Action:
- SQS:DeleteMessage
- SQS:GetQueueAttributes
- SQS:GetQueueUrl
- SQS:ReceiveMessage
- SQS:SendMessage
Resource: !GetAtt S3Queue.Arn
- Effect: Allow
Principal:
AWS: "*"
Action: SQS:SendMessage
Resource: !GetAtt S3Queue.Arn
Condition:
# we have to construct the ARN from the static bucket name to avoid
# the circular dependency
# https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/
ArnLike:
aws:SourceArn: !Join
- ""
- - "arn:aws:s3:::"
- !Join
- "-"
- - !Ref "AWS::Region"
- !Ref "AWS::StackName"
- bucket
Queues:
- Ref: S3Queue
Outputs:
QueueURL:
Description: "URL of new Amazon SQS Queue"
Value:
Ref: S3Queue
QueueARN:
Description: "ARN of new AmazonSQS Queue"
Value:
Fn::GetAtt: S3Queue.Arn
QueueName:
Description: "Name of new Amazon SQS Queue"
Value:
Fn::GetAtt: S3Queue.QueueName
BucketName:
Description: "Name of new Amazon S3 Bucket"
Value:
Ref: S3Bucket
BucketURL:
Description: "URL of new Amazon S3 Bucket"
Value:
Fn::GetAtt: S3Bucket.DomainName
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment