Workaround for 'awsvpc' network-mode containers and multiple target groups from https://github.com/aws/amazon-ecs-agent/issues/1351
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AWSTemplateFormatVersion: 2010-09-09 | |
Description: | | |
Replicates (de)registration events from a source target group to a destionation target group | |
in order to workaround ECS limitation where a single container cannot automatically register | |
to multiple target groups | |
Parameters: | |
SourceTargetGroupArn: | |
Description: ARN of a source target group to replicate targets from | |
Type: String | |
DestinationTargetGroupArn: | |
Description: ARN of target group to replicate targets to | |
Type: String | |
Outputs: | |
AutoUpdateTargetGroupLambdaName: | |
Value: !Ref AutoUpdateTargetGroupLambda | |
AutoUpdateTargetGroupLambdaArn: | |
Value: !GetAtt AutoUpdateTargetGroupLambda.Arn | |
Resources: | |
TargetGroupRegistrationEvents: | |
Type: AWS::Events::Rule | |
Properties: | |
Description: Watch for (De)Register events from source target group | |
State: ENABLED | |
EventPattern: | |
source: | |
- aws.elasticloadbalancing | |
detail-type: | |
- AWS API Call via CloudTrail | |
detail: | |
eventSource: | |
- elasticloadbalancing.amazonaws.com | |
eventName: | |
- RegisterTargets | |
- DeregisterTargets | |
requestParameters: | |
targetGroupArn: | |
- !Ref SourceTargetGroupArn | |
Targets: | |
- Arn: !GetAtt AutoUpdateTargetGroupLambda.Arn | |
Id: lambda | |
AutoUpdateTargetLambdaInvokePermission: | |
Type: AWS::Lambda::Permission | |
Properties: | |
Action: "lambda:InvokeFunction" | |
FunctionName: !GetAtt AutoUpdateTargetGroupLambda.Arn | |
Principal: events.amazonaws.com | |
SourceArn: !GetAtt TargetGroupRegistrationEvents.Arn | |
AutoUpdateTargetGroupLambdaRole: | |
Type: AWS::IAM::Role | |
Properties: | |
AssumeRolePolicyDocument: | |
Version: 2012-10-17 | |
Statement: | |
- Effect: Allow | |
Principal: | |
Service: lambda.amazonaws.com | |
Action: "sts:AssumeRole" | |
ManagedPolicyArns: | |
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" | |
Policies: | |
- PolicyName: AutoUpdateTargetGroupMembers | |
PolicyDocument: | |
Version: 2012-10-17 | |
Statement: | |
- Effect: Allow | |
Action: | |
- "elasticloadbalancing:RegisterTargets" | |
- "elasticloadbalancing:DeregisterTargets" | |
Resource: !Ref DestinationTargetGroupArn | |
Path: /service-role/ | |
AutoUpdateTargetGroupLambda: | |
Type: AWS::Lambda::Function | |
Properties: | |
Description: Automatically (De)Registers destination target group(s) when a source target group changes | |
Environment: | |
Variables: | |
DEST_ARN: !Ref DestinationTargetGroupArn | |
Handler: index.lambda_handler | |
MemorySize: 128 | |
Role: !GetAtt AutoUpdateTargetGroupLambdaRole.Arn | |
Runtime: python3.6 | |
Timeout: 10 | |
Code: | |
ZipFile: | | |
import boto3 | |
import logging | |
import os | |
DEST_ARN = os.environ["DEST_ARN"] | |
def lambda_handler(event, context): | |
""" | |
Given the CloudWatch events 'RegisterTargets' or 'DeregisterTargets', | |
replay (almost) the same events on another target group | |
""" | |
# Initialize logger | |
logger = logging.getLogger() | |
logger.setLevel(logging.INFO) | |
logger.debug(os.environ) | |
logger.debug(event) | |
logger.debug(context) | |
# Instantiate API client for load balancers | |
client = boto3.client("elbv2") | |
# Gather relevant info from CloudWatch event | |
event_name = event["detail"]["eventName"] | |
event_targets = event["detail"]["requestParameters"]["targets"] | |
# Log the relevant information from the received event | |
logger.info("Received CloudWatch Event '%s' with the following targets:" % event_name) | |
logger.info(event_targets) | |
# Reformat / generalize targets | |
targets = list(map( | |
lambda target: {"Id": target["id"]}, | |
event_targets | |
)) | |
# Register new targets | |
if event_name == "RegisterTargets": | |
response = client.register_targets( | |
TargetGroupArn=DEST_ARN, | |
Targets=targets | |
) | |
# Or Deregister old targets | |
elif event_name == "DeregisterTargets": | |
response = client.deregister_targets( | |
TargetGroupArn=DEST_ARN, | |
Targets=targets | |
) | |
# Raise an exception if the Lambda received an unanticipated event | |
else: | |
raise ValueError("Unrecognized event type") | |
return { | |
"statusCode": 200, | |
"body": { | |
"event_processed": event_name, | |
"response": response | |
} | |
} |
For any who find this gist in the future, as of July 30 2019 AWS ECS now has this feature built-in: https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-services-now-support-multiple-load-balancer-target-groups/ .
This solution is still useful in 2022 if your ECS service needs more than 5 allowed target groups per service for NLB.
One advice – turn on (enable) AWS CloudTrail before using this solution, overwise de/register events won't be invoked and your Lambda will never be called. It is sufficient to enable it in only one region with a multi-region setting ON.
Thanks for the awesome work @arpaulnet !
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This was only tested with
awsvpc
ECS tasks andip
(notinstance
) target groups. It works by capturing the register / deregister events from one target group then registering the same target on another target group which has a different port already configured. The cloudformation template will not bring the target groups up-to-date on instantiation, so either manually register the initial targets or re-register the targets by temporarily killing the appropriate service tasks.