ECS Container Instance scale-in protector

README.md

ECS Container Instance scale-in protector

Protect container instance with containers running from scale-in. Uses aws-cli set-instance-protection. Inspired by: https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations

Ignores ecs-agent and dd-agent when counting running containers. You can add more in containers_running in the script below.

Prerequisites

• awk
• awscli
• run as root
• cron (opt.)
• Permissions for EC2 IAM role
  • autoscaling:DescribeAutoScalingInstances
  • autoscaling:SetInstanceProtection

Example crontab

*/5 * * * * root /usr/local/bin/scale_in_protection.sh

ecs_instance_scalein_protector.sh

#!/bin/bash

# inspired by https://stackoverflow.com/questions/45020323/ecs-asg-scaling-down-policy-recommendations
# needs awk, awscli, root, IAM autoscaling:DescribeAutoScalingInstances, SetInstanceProtection

echo "--------------------------------"
echo "[$(date)] Starting script "

set -x
## self EC2 instance ID
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

## self region
az=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
region=${az:0:${#az} - 1}

## self ASG ID
asg_name=$(aws autoscaling describe-auto-scaling-instances --region $region --instance-ids $instance_id | jq .AutoScalingInstances[].AutoScalingGroupName -r)

## number of containers running (except ecs-agent and dd-agent)
containers_running=$(docker ps | awk '{print $2}' | tail -n+2 | grep -v "amazon/amazon-ecs-agent" | grep -v "datadog/docker-dd-agent" | wc -l)

## current protection state
scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ${region} --instance-ids ${instance_id} | jq '.AutoScalingInstances[].ProtectedFromScaleIn' -r)
# scale_protection=$(aws autoscaling describe-auto-scaling-instances --region ap-northeast-1 --instance-ids i-0d24aa0f9d1b5558f --output text | awk '{print tolower($8)}')

if [ ${containers_running} -ge 1 ]; then
    if [ ${scale_protection} == "true" ]; then
        echo "Containers are running and scale protection is on. Doing nothing."
    else
        aws autoscaling set-instance-protection --region ${region} --instance-ids ${instance_id} --auto-scaling-group-name ${asg_name} --protected-from-scale-in
        echo "Containers are running and scale protection is off. Enabling scale-in protection"
    fi
elif  [ ${containers_running} == 0 ]; then
    if [ ${scale_protection} == "true" ]; then
        echo "No containers running and scale protection is on. Disabling scale-in protection."
        aws autoscaling set-instance-protection --region ${region} --instance-ids ${instance_id} --auto-scaling-group-name ${asg_name} --no-protected-from-scale-in
    else
        echo "No containers running and scale protection is off. Doing nothing."
    fi
fi

set +x

echo "[$(date)] Script finished running successfully."
echo "--------------------------------"
echo