Skip to content

Instantly share code, notes, and snippets.

@alexcasalboni
Created February 26, 2019 09:02
Show Gist options
  • Save alexcasalboni/83fa7d8c884339e3365e722827444667 to your computer and use it in GitHub Desktop.
Save alexcasalboni/83fa7d8c884339e3365e722827444667 to your computer and use it in GitHub Desktop.
Amazon Kinesis Data Firehose - AWS Lambda data transformation app (YAML)
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
DeliveryBucket:
Type: AWS::S3::Bucket
StreamProcessFunction:
Type: AWS::Serverless::Function
Properties:
Handler: handler.lambda_handler
Runtime: python2.7
# ...
# all the other properties here
# ...
DeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
DependsOn:
- DeliveryStreamPolicy
Properties:
DeliveryStreamType: DirectPut
ExtendedS3DestinationConfiguration:
Prefix: firehose/
BucketARN: !GetAtt DeliveryBucket.Arn
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 10
CompressionFormat: GZIP
RoleARN: !GetAtt DeliveryStreamRole.Arn
ProcessingConfiguration:
Enabled: true
Processors:
- Type: Lambda
Parameters:
- ParameterName: LambdaArn
ParameterValue: !GetAtt StreamProcessFunction.Arn
DeliveryStreamRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: 'firehose-assume-role'
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
Condition:
StringEquals:
'sts:ExternalId': !Ref 'AWS::AccountId'
DeliveryStreamPolicy:
Type: AWS::IAM::Policy
Properties:
Roles:
- !Ref DeliveryStreamRole
PolicyName: firehose_delivery_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Resource:
- !GetAtt DeliveryBucket.Arn
- !Join
- ''
- - 'arn:aws:s3:::'
- !Ref DeliveryBucket
- '*'
- Effect: Allow
Action:
- 'lambda:InvokeFunction'
- 'lambda:GetFunctionConfiguration'
Resource: !GetAtt StreamProcessFunction.Arn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment