simple_stack_with_lambda.md

AWSTemplateFormatVersion: '2010-09-09' Description: AWS S3 data lake stack. Parameters:

SourceDataBucketName: Description: Data lake bucket with source data files. Type: String Default: datalake.staging.aws

Resources:

DatalakeBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain Properties: BucketName: # !Sub '${DatalakeBucket}' Ref: SourceDataBucketName PublicAccessBlockConfiguration: BlockPublicAcls: true IgnorePublicAcls: true BlockPublicPolicy: true RestrictPublicBuckets: true

Data pipeline orchestrator

OrchestratorLambda: Type: AWS::Lambda::Function DeletionPolicy: Delete DependsOn: OrchestratorLambdaPolicy Properties: FunctionName: pipeline-orchestrator Handler: pipeline_orchestrator/app.lambda_handler Description: Microservice that orchestrates ETL and data loading from AWS S3 to data warehouse. Environment: Variables: DEBUG: true Role: !GetAtt OrchestratorLambdaRole.Arn #arn:aws:iam::868393081606:role/my-lambda-role Code: S3Bucket: orchestrator-lambda.code.aws S3Key: pipeline_orchestrator/stack.zip # S3Key: # Ref: StackPackageS3Key Runtime: python3.8 Timeout: 300 MemorySize: 128

we will need a security role to create a Lambda

OrchestratorLambdaRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - "lambda.amazonaws.com" Action: - "sts:AssumeRole"

OrchestratorLambdaPolicy: Type: AWS::IAM::Policy DependsOn: OrchestratorLambdaRole Properties: Roles: - !Ref OrchestratorLambdaRole PolicyName: 'pipeline-orchestrator-lambda-policy' PolicyDocument: { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Action": "s3:", "Resource": "" }, { "Effect": "Allow", "Action": [ "lambda:" ], "Resource": [ "" ] }, { "Effect": "Allow", "Action": [ "logs:" ], "Resource": "" } ] }