Created
July 16, 2020 22:16
-
-
Save jasonge27/c04c6c2fbb275965a7b9db886c8f9b80 to your computer and use it in GitHub Desktop.
A minimal & economy AWS setup for metaflow (spot instance for batch, no sagemaker, no multi-region RDS)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| AWSTemplateFormatVersion: '2010-09-09' | |
| Description: Stack for complete deployment of Metaflow | |
| Parameters: | |
| VPCCidr: | |
| Type: String | |
| Default: 10.20.0.0/16 | |
| Description: 'CIDR for the Metaflow VPC' | |
| Subnet1Cidr: | |
| Type: String | |
| Default: 10.20.0.0/24 | |
| Description: 'CIDR for Metaflow VPC Subnet 1' | |
| Subnet2Cidr: | |
| Type: String | |
| Default: 10.20.1.0/24 | |
| Description: 'CIDR for Metaflow VPC Subnet 2' | |
| MaxVCPUBatch: | |
| Type: Number | |
| Default: 96 | |
| AllowedValues: [2,4,8,16,32,48,64,96] | |
| MinValue: 16 | |
| MaxValue: 96 | |
| Description: 'Maximum VCPUs for Batch Compute Environment [16-96]' | |
| MinVCPUBatch: | |
| Type: Number | |
| Default: 0 | |
| MinValue: 0 | |
| MaxValue: 16 | |
| AllowedValues: [0, 2,4,8,16] | |
| Description: 'Minimum VCPUs for Batch Compute Environment [0-16]' | |
| DesiredVCPUBatch: | |
| Type: Number | |
| Default: 0 | |
| MinValue: 0 | |
| MaxValue: 16 | |
| AllowedValues: [0, 2,4,8,16] | |
| Description: 'Desired Starting VCPUs for Batch Compute Environment [0-16]' | |
| CustomRole: | |
| Type: String | |
| Default: 'false' | |
| AllowedValues: ['false', 'true'] | |
| Description: 'Enable custom role with restricted permissions?' | |
| APIBasicAuth: | |
| Type: String | |
| Default: 'false' | |
| AllowedValues: ['false', 'true'] | |
| Description: 'Enable basic auth for API Gateway? (requires key export)' | |
| Mappings: | |
| ServiceInfo: | |
| StackName: | |
| value: 'metaflow-infrastructure' | |
| ServiceName: | |
| value: 'metadata-service' | |
| ImageUrl: | |
| value: 'netflixoss/metaflow_metadata_service' | |
| ContainerPort: | |
| value: 8080 | |
| ContainerCpu: | |
| value: 512 | |
| ContainerMemory: | |
| value: 1024 | |
| Path: | |
| value: '*' | |
| Priority: | |
| value: 1 | |
| DesiredCount: | |
| value: 1 | |
| Role: | |
| value: "" | |
| Conditions: | |
| EnableAuth: !Equals [ !Ref APIBasicAuth, 'true'] | |
| EnableRole: !Equals [ !Ref CustomRole, 'true'] | |
| Resources: | |
| VPC: | |
| Type: AWS::EC2::VPC | |
| Properties: | |
| EnableDnsSupport: true | |
| EnableDnsHostnames: true | |
| CidrBlock: !Ref VPCCidr | |
| Subnet1: | |
| Type: AWS::EC2::Subnet | |
| Properties: | |
| AvailabilityZone: | |
| Fn::Select: | |
| - 0 | |
| - Fn::GetAZs: {Ref: 'AWS::Region'} | |
| VpcId: !Ref 'VPC' | |
| CidrBlock: !Ref Subnet1Cidr | |
| MapPublicIpOnLaunch: true | |
| Subnet2: | |
| Type: AWS::EC2::Subnet | |
| Properties: | |
| AvailabilityZone: | |
| Fn::Select: | |
| - 1 | |
| - Fn::GetAZs: {Ref: 'AWS::Region'} | |
| VpcId: !Ref 'VPC' | |
| CidrBlock: !Ref Subnet2Cidr | |
| MapPublicIpOnLaunch: true | |
| InternetGateway: | |
| Type: AWS::EC2::InternetGateway | |
| GatewayAttachement: | |
| Type: AWS::EC2::VPCGatewayAttachment | |
| Properties: | |
| VpcId: !Ref 'VPC' | |
| InternetGatewayId: !Ref 'InternetGateway' | |
| PublicRouteTable: | |
| Type: AWS::EC2::RouteTable | |
| Properties: | |
| VpcId: !Ref 'VPC' | |
| DefaultGateway: | |
| Type: AWS::EC2::Route | |
| DependsOn: GatewayAttachement | |
| Properties: | |
| RouteTableId: !Ref 'PublicRouteTable' | |
| DestinationCidrBlock: '0.0.0.0/0' | |
| GatewayId: !Ref 'InternetGateway' | |
| Subnet1RTA: | |
| Type: AWS::EC2::SubnetRouteTableAssociation | |
| Properties: | |
| SubnetId: !Ref Subnet1 | |
| RouteTableId: !Ref PublicRouteTable | |
| Subnet2RTA: | |
| Type: AWS::EC2::SubnetRouteTableAssociation | |
| Properties: | |
| SubnetId: !Ref Subnet2 | |
| RouteTableId: !Ref PublicRouteTable | |
| ECSCluster: | |
| Type: AWS::ECS::Cluster | |
| Properties: | |
| ClusterSettings: | |
| - Name: containerInsights | |
| Value: enabled | |
| FargateSecurityGroup: | |
| Type: AWS::EC2::SecurityGroup | |
| Properties: | |
| GroupDescription: Security Group for Fargate | |
| VpcId: !Ref 'VPC' | |
| NLBIngressRule: | |
| Type: 'AWS::EC2::SecurityGroupIngress' | |
| Properties: | |
| Description: 'Allow API Calls Internally' | |
| GroupId: !Ref FargateSecurityGroup | |
| IpProtocol: tcp | |
| FromPort: 8080 | |
| ToPort: 8080 | |
| CidrIp: !GetAtt 'VPC.CidrBlock' | |
| FargateInternalRule: | |
| Type: AWS::EC2::SecurityGroupIngress | |
| Properties: | |
| Description: Internal Communication | |
| GroupId: !Ref 'FargateSecurityGroup' | |
| IpProtocol: -1 | |
| SourceSecurityGroupId: !Ref 'FargateSecurityGroup' | |
| NLB: | |
| Type: AWS::ElasticLoadBalancingV2::LoadBalancer | |
| Properties: | |
| Scheme: internal | |
| Type: network | |
| Subnets: | |
| - !Ref Subnet1 | |
| - !Ref Subnet2 | |
| NLBListener: | |
| Type: AWS::ElasticLoadBalancingV2::Listener | |
| DependsOn: | |
| - NLB | |
| Properties: | |
| DefaultActions: | |
| - TargetGroupArn: !Ref 'NLBTargetGroup' | |
| Type: 'forward' | |
| LoadBalancerArn: !Ref 'NLB' | |
| Port: 80 | |
| Protocol: TCP | |
| ECSRole: | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Statement: | |
| - Effect: Allow | |
| Principal: | |
| Service: [ecs.amazonaws.com] | |
| Action: ['sts:AssumeRole'] | |
| Path: / | |
| Policies: | |
| - PolicyName: ecs-service | |
| PolicyDocument: | |
| Statement: | |
| - Effect: Allow | |
| Action: | |
| - 'ec2:AttachNetworkInterface' | |
| - 'ec2:CreateNetworkInterface' | |
| - 'ec2:CreateNetworkInterfacePermission' | |
| - 'ec2:DeleteNetworkInterface' | |
| - 'ec2:DeleteNetworkInterfacePermission' | |
| - 'ec2:Describe*' | |
| - 'ec2:DetachNetworkInterface' | |
| - 'elasticloadbalancing:DeregisterInstancesFromLoadBalancer' | |
| - 'elasticloadbalancing:DeregisterTargets' | |
| - 'elasticloadbalancing:Describe*' | |
| - 'elasticloadbalancing:RegisterInstancesWithLoadBalancer' | |
| - 'elasticloadbalancing:RegisterTargets' | |
| Resource: '*' | |
| ECSTaskExecutionRole: | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Statement: | |
| - Effect: Allow | |
| Principal: | |
| Service: [ecs-tasks.amazonaws.com] | |
| Action: ['sts:AssumeRole'] | |
| Path: / | |
| Policies: | |
| - PolicyName: AmazonECSTaskExecutionRolePolicy | |
| PolicyDocument: | |
| Statement: | |
| - Effect: Allow | |
| Action: | |
| - 'ecr:GetAuthorizationToken' | |
| - 'ecr:BatchCheckLayerAvailability' | |
| - 'ecr:GetDownloadUrlForLayer' | |
| - 'ecr:BatchGetImage' | |
| - 'logs:CreateLogStream' | |
| - 'logs:PutLogEvents' | |
| Resource: '*' | |
| MetadataSvcECSTaskRole: | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Effect: Allow | |
| Principal: | |
| Service: | |
| - ecs-tasks.amazonaws.com | |
| Action: | |
| - sts:AssumeRole | |
| Path: / | |
| Policies: | |
| - PolicyName: CustomS3Batch | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: ObjectAccessMetadataService | |
| Effect: Allow | |
| Action: | |
| - s3:GetObject | |
| Resource: !Join ['', [ !GetAtt 'MetaflowS3Bucket.Arn', '/*' ]] | |
| - PolicyName: DenyPresignedBatch | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: DenyPresignedBatch | |
| Effect: Deny | |
| Action: s3:* | |
| Resource: '*' | |
| Condition: | |
| StringNotEquals: | |
| s3:authType: REST-HEADER | |
| TaskDefinition: | |
| Type: AWS::ECS::TaskDefinition | |
| Properties: | |
| Family: !FindInMap ['ServiceInfo', 'ServiceName', 'value'] | |
| Cpu: !FindInMap ['ServiceInfo', 'ContainerCpu', 'value'] | |
| Memory: !FindInMap ['ServiceInfo', 'ContainerMemory', 'value'] | |
| NetworkMode: awsvpc | |
| RequiresCompatibilities: | |
| - FARGATE | |
| ExecutionRoleArn: !Ref 'ECSTaskExecutionRole' | |
| TaskRoleArn: !GetAtt 'MetadataSvcECSTaskRole.Arn' | |
| ContainerDefinitions: | |
| - Name: !FindInMap ['ServiceInfo', 'ServiceName', 'value'] | |
| Environment: | |
| - Name: "MF_METADATA_DB_HOST" | |
| Value: !GetAtt 'RDSMasterInstance.Endpoint.Address' | |
| - Name: "MF_METADATA_DB_PORT" | |
| Value: "5432" | |
| - Name: "MF_METADATA_DB_USER" | |
| Value: "master" | |
| - Name: "MF_METADATA_DB_PSWD" | |
| Value: !Join ['', ['{{resolve:secretsmanager:', !Ref MyRDSSecret, ':SecretString:password}}' ]] | |
| - Name: "MF_METADATA_DB_NAME" | |
| Value: "metaflow" | |
| Cpu: !FindInMap ['ServiceInfo', 'ContainerCpu', 'value'] | |
| Memory: !FindInMap ['ServiceInfo', 'ContainerMemory', 'value'] | |
| Image: !FindInMap ['ServiceInfo', 'ImageUrl', 'value'] | |
| PortMappings: | |
| - ContainerPort: !FindInMap ['ServiceInfo', 'ContainerPort', 'value'] | |
| ECSFargateService: | |
| Type: AWS::ECS::Service | |
| Properties: | |
| ServiceName: !FindInMap ['ServiceInfo', 'ServiceName', 'value'] | |
| Cluster: !Ref 'ECSCluster' | |
| LaunchType: FARGATE | |
| DeploymentConfiguration: | |
| MaximumPercent: 200 | |
| MinimumHealthyPercent: 75 | |
| DesiredCount: !FindInMap ['ServiceInfo', 'DesiredCount', 'value'] | |
| NetworkConfiguration: | |
| AwsvpcConfiguration: | |
| AssignPublicIp: ENABLED | |
| SecurityGroups: | |
| - !Ref 'FargateSecurityGroup' | |
| Subnets: | |
| - !Ref 'Subnet1' | |
| - !Ref 'Subnet2' | |
| TaskDefinition: !Ref 'TaskDefinition' | |
| LoadBalancers: | |
| - ContainerName: !FindInMap ['ServiceInfo', 'ServiceName', 'value'] | |
| ContainerPort: !FindInMap ['ServiceInfo', 'ContainerPort', 'value'] | |
| TargetGroupArn: !Ref 'NLBTargetGroup' | |
| NLBTargetGroup: | |
| Type: AWS::ElasticLoadBalancingV2::TargetGroup | |
| Properties: | |
| HealthCheckIntervalSeconds: 10 | |
| HealthCheckProtocol: TCP | |
| HealthCheckTimeoutSeconds: 10 | |
| HealthyThresholdCount: 2 | |
| TargetType: ip | |
| Port: !FindInMap ['ServiceInfo', 'ContainerPort', 'value'] | |
| Protocol: TCP | |
| UnhealthyThresholdCount: 2 | |
| VpcId: !Ref 'VPC' | |
| DBSubnetGroup: | |
| Type: AWS::RDS::DBSubnetGroup | |
| Properties: | |
| DBSubnetGroupDescription: DBSubnetGroup for RDS instances | |
| SubnetIds: | |
| - Ref: Subnet1 | |
| - Ref: Subnet2 | |
| RDSSecurityGroup: | |
| Type: AWS::EC2::SecurityGroup | |
| Properties: | |
| GroupDescription: Security Group for RDS | |
| VpcId: !Ref 'VPC' | |
| PostgresIngressRule: | |
| Type: AWS::EC2::SecurityGroupIngress | |
| Properties: | |
| GroupId: !Ref 'RDSSecurityGroup' | |
| SourceSecurityGroupId: !Ref 'FargateSecurityGroup' | |
| IpProtocol: tcp | |
| FromPort: 5432 | |
| ToPort: 5432 | |
| RDSMasterInstance: | |
| Type: AWS::RDS::DBInstance | |
| Properties: | |
| DBName: 'metaflow' | |
| AllocatedStorage: 20 | |
| DBInstanceClass: 'db.t2.micro' | |
| DeleteAutomatedBackups: 'true' | |
| StorageType: 'gp2' | |
| Engine: 'postgres' | |
| EngineVersion: '11.5' | |
| MasterUsername: !Join ['', ['{{resolve:secretsmanager:', !Ref MyRDSSecret, ':SecretString:username}}' ]] | |
| MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref MyRDSSecret, ':SecretString:password}}' ]] | |
| VPCSecurityGroups: | |
| - !Ref 'RDSSecurityGroup' | |
| DBSubnetGroupName: !Ref 'DBSubnetGroup' | |
| MultiAZ: 'false' | |
| MyRDSSecret: | |
| Type: "AWS::SecretsManager::Secret" | |
| Properties: | |
| Description: "This is a Secrets Manager secret for an RDS DB instance" | |
| GenerateSecretString: | |
| SecretStringTemplate: '{"username": "master"}' | |
| GenerateStringKey: "password" | |
| PasswordLength: 16 | |
| ExcludeCharacters: '"@/\' | |
| SecretRDSInstanceAttachment: | |
| Type: "AWS::SecretsManager::SecretTargetAttachment" | |
| Properties: | |
| SecretId: !Ref MyRDSSecret | |
| TargetId: !Ref RDSMasterInstance | |
| TargetType: AWS::RDS::DBInstance | |
| MetaflowS3Bucket: | |
| Type: 'AWS::S3::Bucket' | |
| DeletionPolicy: Retain | |
| Properties: | |
| AccessControl: Private | |
| BucketEncryption: | |
| ServerSideEncryptionConfiguration: | |
| - ServerSideEncryptionByDefault: | |
| SSEAlgorithm: AES256 | |
| PublicAccessBlockConfiguration: | |
| BlockPublicAcls: true | |
| BlockPublicPolicy: true | |
| IgnorePublicAcls: true | |
| RestrictPublicBuckets: true | |
| MetaflowUserRole: | |
| Condition: 'EnableRole' | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Effect: Allow | |
| Principal: | |
| AWS: | |
| - !GetAtt 'MetadataSvcECSTaskRole.Arn' | |
| Action: | |
| - sts:AssumeRole | |
| Path: / | |
| Policies: | |
| - | |
| PolicyName: "Metaflow-Policy" | |
| PolicyDocument: | |
| Version: "2012-10-17" | |
| Statement: | |
| - Effect: "Allow" | |
| Action: | |
| - "cloudformation:DescribeStacks" | |
| - "cloudformation:*Stack" | |
| - "cloudformation:*ChangeSet" | |
| Resource: | |
| - !Sub "arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-*" | |
| - Effect: "Allow" | |
| Action: | |
| - "s3:*Object" | |
| Resource: | |
| - !GetAtt 'MetaflowS3Bucket.Arn' | |
| - !Join ['', [ !GetAtt 'MetaflowS3Bucket.Arn', '/*' ]] | |
| - Effect: "Allow" | |
| Action: | |
| - "iam:PassRole" | |
| Resource: | |
| - !Sub "arn:aws:iam::${AWS::AccountId}:role/${AWS::StackName}-*" | |
| - Effect: "Allow" | |
| Action: | |
| - "kms:Decrypt" | |
| - "kms:Encrypt" | |
| Resource: | |
| - !Sub "aws:aws:kms:${AWS::Region}:${AWS::AccountId}:key/" | |
| - PolicyName: BatchPerms | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| - Sid: JobsPermissions | |
| Effect: Allow | |
| Action: | |
| - "batch:TerminateJob" | |
| - "batch:DescribeJobs" | |
| - "batch:DescribeJobDefinitions" | |
| - "batch:DescribeJobQueues" | |
| - "batch:RegisterJobDefinition" | |
| Resource: '*' | |
| - Sid: DefinitionsPermissions | |
| Effect: Allow | |
| Action: | |
| - "batch:SubmitJob" | |
| Resource: | |
| - !Ref "JobQueue" | |
| - !Sub arn:aws:batch:${AWS::Region}:${AWS::AccountId}:job-definition/*:* | |
| - PolicyName: CustomS3ListAccess | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: BucketAccess | |
| Effect: Allow | |
| Action: s3:ListBucket | |
| Resource: !GetAtt 'MetaflowS3Bucket.Arn' | |
| - PolicyName: LogPerms | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: GetLogs | |
| Effect: Allow | |
| Action: logs:GetLogEvents | |
| Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:*:log-stream:* | |
| RandomString: | |
| Type: AWS::SecretsManager::Secret | |
| Properties: | |
| Description: 'Random String' | |
| GenerateSecretString: | |
| SecretStringTemplate: '{"username": "admin"}' | |
| GenerateStringKey: 'password' | |
| ExcludePunctuation: 'true' | |
| PasswordLength: 8 | |
| ExcludeCharacters: '"@/\' | |
| BatchS3TaskRole: | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Effect: Allow | |
| Principal: | |
| Service: | |
| - ecs-tasks.amazonaws.com | |
| Action: | |
| - sts:AssumeRole | |
| Path: / | |
| Policies: | |
| - PolicyName: CustomS3ListBatch | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: BucketAccessBatch | |
| Effect: Allow | |
| Action: s3:ListBucket | |
| Resource: !GetAtt 'MetaflowS3Bucket.Arn' | |
| - PolicyName: CustomS3Batch | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: ObjectAccessBatch | |
| Effect: Allow | |
| Action: | |
| - s3:PutObject | |
| - s3:GetObject | |
| - s3:DeleteObject | |
| Resource: !Join ['', [ !GetAtt 'MetaflowS3Bucket.Arn', '/*' ]] | |
| - PolicyName: DenyPresignedBatch | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: DenyPresignedBatch | |
| Effect: Deny | |
| Action: s3:* | |
| Resource: '*' | |
| Condition: | |
| StringNotEquals: | |
| s3:authType: REST-HEADER | |
| BatchExecutionRole: | |
| Type: AWS::IAM::Role | |
| Properties: | |
| AssumeRolePolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Effect: Allow | |
| Principal: | |
| Service: | |
| - batch.amazonaws.com | |
| Action: | |
| - sts:AssumeRole | |
| Path: / | |
| Policies: | |
| - PolicyName: IAM_PASS_ROLE | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: VisualEditor0 | |
| Effect: Allow | |
| Action: iam:PassRole | |
| Resource: '*' | |
| Condition: | |
| StringEquals: | |
| iam:PassedToService: | |
| - ec2.amazonaws.com | |
| - ec2.amazonaws.com.cn | |
| - ecs-tasks.amazonaws.com | |
| - PolicyName: IAM_PASS_ROLE_SPOT_FLEET | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: VisualEditor0 | |
| Effect: Allow | |
| Action: iam:PassRole | |
| Resource: "arn:aws:iam::*:role/aws-ec2-spot-fleet-tagging-role" | |
| - PolicyName: custom_access_policy | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: VisualEditor2 | |
| Effect: Allow | |
| Action: | |
| - ec2:DescribeAccountAttributes | |
| - ec2:DescribeInstances | |
| - ec2:DescribeInstanceAttribute | |
| - ec2:DescribeSubnets | |
| - ec2:DescribeSecurityGroups | |
| - ec2:DescribeKeyPairs | |
| - ec2:DescribeImages | |
| - ec2:DescribeImageAttribute | |
| - ec2:DescribeSpotInstanceRequests | |
| - ec2:DescribeSpotFleetInstances | |
| - ec2:DescribeSpotFleetRequests | |
| - ec2:DescribeSpotPriceHistory | |
| - ec2:DescribeVpcClassicLink | |
| - ec2:DescribeLaunchTemplateVersions | |
| - ec2:CreateLaunchTemplate | |
| - ec2:DeleteLaunchTemplate | |
| - ec2:RequestSpotFleet | |
| - ec2:CancelSpotFleetRequests | |
| - ec2:ModifySpotFleetRequest | |
| - ec2:TerminateInstances | |
| - ec2:RunInstances | |
| - ec2:CreateTags | |
| - ec2:DescribeSpotFleetRequestHistory | |
| - autoscaling:DescribeAccountLimits | |
| - autoscaling:DescribeAutoScalingGroups | |
| - autoscaling:DescribeLaunchConfigurations | |
| - autoscaling:DescribeAutoScalingInstances | |
| - autoscaling:CreateLaunchConfiguration | |
| - autoscaling:CreateAutoScalingGroup | |
| - autoscaling:UpdateAutoScalingGroup | |
| - autoscaling:SetDesiredCapacity | |
| - autoscaling:DeleteLaunchConfiguration | |
| - autoscaling:DeleteAutoScalingGroup | |
| - autoscaling:CreateOrUpdateTags | |
| - autoscaling:SuspendProcesses | |
| - autoscaling:PutNotificationConfiguration | |
| - autoscaling:TerminateInstanceInAutoScalingGroup | |
| - ecs:DescribeClusters | |
| - ecs:DescribeContainerInstances | |
| - ecs:DescribeTaskDefinition | |
| - ecs:DescribeTasks | |
| - ecs:ListClusters | |
| - ecs:ListContainerInstances | |
| - ecs:ListTaskDefinitionFamilies | |
| - ecs:ListTaskDefinitions | |
| - ecs:ListTasks | |
| - ecs:CreateCluster | |
| - ecs:DeleteCluster | |
| - ecs:RegisterTaskDefinition | |
| - ecs:DeregisterTaskDefinition | |
| - ecs:RunTask | |
| - ecs:StartTask | |
| - ecs:StopTask | |
| - ecs:UpdateContainerAgent | |
| - ecs:DeregisterContainerInstance | |
| - logs:CreateLogGroup | |
| - logs:CreateLogStream | |
| - logs:PutLogEvents | |
| - logs:DescribeLogGroups | |
| - iam:GetInstanceProfile | |
| - iam:GetRole | |
| - iam:ListRoles | |
| - iam:ListInstanceProfiles | |
| Resource: '*' | |
| - PolicyName: iam_custom_policies | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: VisualEditor3 | |
| Effect: Allow | |
| Action: iam:CreateServiceLinkedRole | |
| Resource: '*' | |
| Condition: | |
| StringEquals: | |
| iam:AWSServiceName: | |
| - autoscaling.amazonaws.com | |
| - ecs.amazonaws.com | |
| - PolicyName: ec2_custom_policies | |
| PolicyDocument: | |
| Version: '2012-10-17' | |
| Statement: | |
| Sid: VisualEditor4 | |
| Effect: Allow | |
| Action: ec2:CreateTags | |
| Resource: '*' | |
| Condition: | |
| StringEquals: | |
| ec2:CreateAction: RunInstances | |
| ComputeEnvironmentSecurityGroup: | |
| Type: "AWS::EC2::SecurityGroup" | |
| Properties: | |
| GroupDescription: 'Security Group for Compute Environment' | |
| VpcId: !Ref VPC | |
| SecurityGroupIngress: | |
| - CidrIp: "0.0.0.0/0" | |
| IpProtocol: "TCP" | |
| FromPort: 8080 | |
| ToPort: 8080 | |
| - CidrIp: "0.0.0.0/0" | |
| IpProtocol: "SSH" | |
| FromPort: 22 | |
| ToPort: 22 | |
| ComputeEnvironment: | |
| DependsOn: ECSRole | |
| Type: AWS::Batch::ComputeEnvironment | |
| Properties: | |
| Type: MANAGED | |
| ServiceRole: !GetAtt 'BatchExecutionRole.Arn' | |
| ComputeEnvironmentName: !Join [ "-", [ 'batch-compute-env', !Ref 'AWS::StackName' ] ] | |
| ComputeResources: | |
| SecurityGroupIds: | |
| - !Ref ComputeEnvironmentSecurityGroup | |
| Type: SPOT | |
| AllocationStrategy: BEST_FIT | |
| Subnets: | |
| - !Ref Subnet1 | |
| - !Ref Subnet2 | |
| MinvCpus: !Ref MinVCPUBatch | |
| MaxvCpus: !Ref MaxVCPUBatch | |
| InstanceRole: ecsInstanceRole | |
| Ec2KeyPair: metaflow-batch | |
| SpotIamFleetRole: arn:aws:iam::079748302414:role/aws-ec2-spot-fleet-tagging-role | |
| InstanceTypes: | |
| - m5d | |
| - p2.xlarge | |
| DesiredvCpus: !Ref DesiredVCPUBatch | |
| State: ENABLED | |
| JobQueue: | |
| DependsOn: ComputeEnvironment | |
| Type: AWS::Batch::JobQueue | |
| Properties: | |
| ComputeEnvironmentOrder: | |
| - Order: 1 | |
| ComputeEnvironment: !Join [ "-", [ 'batch-compute-env', !Ref 'AWS::StackName' ] ] | |
| State: ENABLED | |
| Priority: 1 | |
| JobQueueName: !Join [ "-", [ 'job-queue', !Ref 'AWS::StackName' ] ] | |
| Api: | |
| DependsOn: VpcLink | |
| Type: 'AWS::ApiGateway::RestApi' | |
| Properties: | |
| Name: !Join ['-', [!Ref 'AWS::StackName', 'api'] ] | |
| ApiResource: | |
| Type: 'AWS::ApiGateway::Resource' | |
| Properties: | |
| ParentId: !GetAtt Api.RootResourceId | |
| RestApiId: !Ref Api | |
| PathPart: '{proxy+}' | |
| VpcLink: | |
| Type: AWS::ApiGateway::VpcLink | |
| Properties: | |
| Name: !Join ['-', [!Ref 'AWS::StackName', 'vpclink'] ] | |
| TargetArns: | |
| - !Ref NLB | |
| ProxyMethod: | |
| Type: 'AWS::ApiGateway::Method' | |
| Properties: | |
| HttpMethod: ANY | |
| ApiKeyRequired: !If [ EnableAuth, 'true', !Ref "AWS::NoValue" ] | |
| ResourceId: !Ref ApiResource | |
| RestApiId: !Ref Api | |
| AuthorizationType: NONE | |
| RequestParameters: | |
| method.request.path.proxy: true | |
| Integration: | |
| ConnectionType: VPC_LINK | |
| ConnectionId: !Ref VpcLink | |
| CacheKeyParameters: | |
| - 'method.request.path.proxy' | |
| RequestParameters: | |
| integration.request.path.proxy: 'method.request.path.proxy' | |
| IntegrationHttpMethod: ANY | |
| Type: HTTP_PROXY | |
| Uri: !Join ['', ['http://', !GetAtt 'NLB.DNSName', '/{proxy}'] ] | |
| PassthroughBehavior: WHEN_NO_MATCH | |
| IntegrationResponses: | |
| - StatusCode: 200 | |
| ApiDeployment: | |
| DependsOn: | |
| - ProxyMethod | |
| Type: 'AWS::ApiGateway::Deployment' | |
| Properties: | |
| RestApiId: !Ref Api | |
| StageName: api | |
| ApiKey: | |
| Condition: 'EnableAuth' | |
| Type: 'AWS::ApiGateway::ApiKey' | |
| DependsOn: | |
| - Api | |
| - ApiDeployment | |
| Properties: | |
| Name: !Join ['-', [!Ref 'AWS::StackName', ApiKey] ] | |
| Enabled: 'true' | |
| ApiUsagePlan: | |
| Condition: 'EnableAuth' | |
| Type: "AWS::ApiGateway::UsagePlan" | |
| DependsOn: | |
| - Api | |
| - ApiDeployment | |
| Properties: | |
| ApiStages: | |
| - ApiId: !Ref Api | |
| Stage: api | |
| UsagePlanName: !Join ["", [{"Ref": "AWS::StackName"}, "-usage-plan"]] | |
| ApiUsagePlanKey: | |
| Condition: 'EnableAuth' | |
| Type: "AWS::ApiGateway::UsagePlanKey" | |
| DependsOn: | |
| - Api | |
| - ApiDeployment | |
| Properties: | |
| KeyId: !Ref ApiKey | |
| KeyType: API_KEY | |
| UsagePlanId: !Ref ApiUsagePlan | |
| Outputs: | |
| MetaflowDataStoreS3Url: | |
| Description: Amazon S3 URL for Metaflow DataStore [METAFLOW_DATASTORE_SYSROOT_S3] | |
| Value: !Sub "s3://${MetaflowS3Bucket}/metaflow" | |
| MetaflowDataToolsS3Url: | |
| Description: Amazon S3 URL for Metaflow DataTools [METAFLOW_DATATOOLS_S3ROOT] | |
| Value: !Sub "s3://${MetaflowS3Bucket}/data" | |
| BatchJobQueueArn: | |
| Description: AWS Batch Job Queue ARN for Metaflow [METAFLOW_BATCH_JOB_QUEUE] | |
| Value: !Ref JobQueue | |
| ECSJobRoleForBatchJobs: | |
| Description: Role for AWS Batch to Access Amazon S3 [METAFLOW_ECS_S3_ACCESS_IAM_ROLE] | |
| Value: !GetAtt 'BatchS3TaskRole.Arn' | |
| ServiceUrl: | |
| Description: "URL for Metadata Service (Open to Public Access) [METAFLOW_SERVICE_URL]" | |
| Value: !Sub "https://${Api}.execute-api.${AWS::Region}.amazonaws.com/api/" | |
| InternalServiceUrl: | |
| Description: "URL for Metadata Service (Accessible in VPC) [METAFLOW_SERVICE_INTERNAL_URL]" | |
| Value: !Sub "http://${NLB.DNSName}/" | |
| ApiKeyId: | |
| Condition: 'EnableAuth' | |
| Description: "API Gateway Key ID for Metadata Service. Fetch Key from AWS Console [METAFLOW_SERVICE_AUTH_KEY]" | |
| Value: !Ref 'ApiKey' | |
| MetaflowUserRoleArn: | |
| Condition: 'EnableRole' | |
| Description: "IAM Role for Metaflow Stack" | |
| Value: !GetAtt "MetaflowUserRole.Arn" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment