Skip to content

Instantly share code, notes, and snippets.

@lizturp
Last active November 3, 2022 03:34
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
  • Save lizturp/33d202470eb95980b33cbdf16e2ea465 to your computer and use it in GitHub Desktop.
Save lizturp/33d202470eb95980b33cbdf16e2ea465 to your computer and use it in GitHub Desktop.
AWS Cloudformation template to build a firehose delivery stream to S3, with a kinesis stream as the source. JSON, but it's fine.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "The AWS CloudFormation template for Kinesis Stream",
"Parameters": {
"Environment": {
"Description": "dev, stage, or prod - this is for bucket tags",
"Type": "String",
"MinLength": "3",
"MaxLength": "5"
}
},
"Resources": {
"KinesisStream": {
"Type" : "AWS::Kinesis::Stream",
"Properties" : {
"Name" : {"Fn::Join" : [ "", [ "test-kinesis-fh-", {"Ref": "Environment"} ] ]},
"RetentionPeriodHours" : 24,
"ShardCount" : 8,
"Tags" : [
{"Key": "Environment", "Value": {"Ref": "Environment"}},
{"Key": "Project", "Value": "Test Kinesis"},
{"Key": "Owner", "Value": "liz"}
]
}
},
"KinesisFirehoseDeliveryStream": {
"Type": "AWS::KinesisFirehose::DeliveryStream",
"Properties": {
"DeliveryStreamName": {"Fn::Join" : [ "", [ "test-kinesis-fh-", {"Ref": "Environment"} ] ]},
"DeliveryStreamType": "KinesisStreamAsSource",
"KinesisStreamSourceConfiguration": {
"KinesisStreamARN": {"Fn::GetAtt" : ["KinesisStream", "Arn"] },
"RoleARN": {"Fn::GetAtt": ["FirehoseDeliveryIAMRole", "Arn"]}
},
"S3DestinationConfiguration": {
"BucketARN": "arn:aws:s3:::test-bucket-name",
"Prefix": "cloudformation-test/kinesis-fh",
"BufferingHints": {
"IntervalInSeconds": 60,
"SizeInMBs": 100
},
"CloudWatchLoggingOptions" : {
"Enabled" : "false"
},
"CompressionFormat" : "GZIP",
"EncryptionConfiguration" : {
"NoEncryptionConfig" : "NoEncryption"
},
"RoleARN": {"Fn::GetAtt": ["FirehoseDeliveryIAMRole", "Arn"]}
}
},
"DependsOn": ["FirehoseDeliveryIAMPolicy"]
},
"FirehoseDeliveryIAMRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "ACCOUNT_NUMBER"
}
}
}]
}
}
},
"FirehoseDeliveryIAMPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": {"Fn::Join" : [ "", [ "test-kinesis-fh-", {"Ref": "Environment"} ] ]},
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::test-bucket-name/cloudformation-test/kinesis-fh*"
]
},
{
"Effect": "Allow",
"Action": [
"kinesis:DescribeStream",
"kinesis:GetShardIterator",
"kinesis:GetRecords"
],
"Resource": {"Fn::GetAtt": ["KinesisStream", "Arn"]}
}]
},
"Roles": [{"Ref": "FirehoseDeliveryIAMRole"}]
},
"DependsOn": ["KinesisStream"]
}
},
"Outputs": {
"kinesisStreamArn": {
"Description": "Kinesis Stream ARN",
"Value": {"Fn::GetAtt": ["KinesisStream", "Arn"]}},
"firehoseDeliveryStreamArn": {
"Description": "Firehose Delivery Stream ARN",
"Value": {"Fn::GetAtt": ["KinesisFirehoseDeliveryStream", "Arn"]}},
"firehoseDeliveryRoleArn": {
"Description": "Firehose Delivery Role ARN",
"Value": {"Fn::GetAtt": ["FirehoseDeliveryIAMRole", "Arn"]}}
}
}
@ashokrayal
Copy link

Hi,
I have used the same code and I am getting this error - Firehose is unable to assume role

@thepont
Copy link

thepont commented Oct 26, 2020

I think you have to change ACCOUNT_NUMBER, to your actual account number.

"FirehoseDeliveryIAMRole": {
        "Type": "AWS::IAM::Role",
        "Properties": {
          "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Sid": "",
                "Effect": "Allow",
                "Principal": {
                  "Service": "firehose.amazonaws.com"
                },
                "Action": "sts:AssumeRole",
                "Condition": {
                  "StringEquals": {
                    "sts:ExternalId": "ACCOUNT_NUMBER"
                  }
                }
              }]
          }
        }
      }

@bors40
Copy link

bors40 commented Jul 21, 2022

Does this code create the s3 bucket itself or you have to create the s3 bucket prior?

@lizturp
Copy link
Author

lizturp commented Jul 28, 2022

Does this code create the s3 bucket itself or you have to create the s3 bucket prior?

No bucket resource is created with this example. Replace line 36 with an existing bucket arn. Also line 94 in the policy.

However, I will say this code is very old. I added this because in late 2017, the AWS docs didn't have a complete example of this configuration and I spent quite a bit of time getting this correct. I figured other's were probably running into the same issues I was, so I shared this out.

If I remember correctly, I had encryption config off because there was some issue back then with the KMS integration. I'm sure that's not the case today.

I should update this one day :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment