AWS ECS and ECR deployment via Docker and Gitlab CI

.gitlab-ci.yml

image: docker:latest

variables:
  REPOSITORY_URL: <AWS ACCOUNT ID>.dkr.ecr.eu-central-1.amazonaws.com/<ECS REPOSITORY NAME>
  REGION: eu-central-1
  TASK_DEFINTION_NAME: <TASK DEFINITION NAME>
  CLUSTER_NAME: <CLUSTER NAME>
  SERVICE_NAME: <SERVICE NAME>

services:
- docker:dind

before_script:
  - apk add --no-cache curl jq python py-pip
  - pip install awscli
  - $(aws ecr get-login --no-include-email --region "${REGION}")
  - IMAGE_TAG="$(echo $CI_COMMIT_SHA | head -c 8)"

stages:
  - build
  - deploy

build:
  stage: build
  script:
    - echo "Building image..."
    - docker build -t $REPOSITORY_URL:latest .
    - echo "Tagging image..."
    - docker tag $REPOSITORY_URL:latest $REPOSITORY_URL:$IMAGE_TAG
    - echo "Pushing image..."
    - docker push $REPOSITORY_URL:latest
    - docker push $REPOSITORY_URL:$IMAGE_TAG
  only:
    - master

deploy:
  stage: deploy
  script:
    - echo $REPOSITORY_URL:$IMAGE_TAG
    - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_DEFINTION_NAME" --region "${REGION}")
    - NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | python $CI_PROJECT_DIR/update_task_definition_image.py $REPOSITORY_URL:$IMAGE_TAG)
    - echo "Registering new container definition..."
    - aws ecs register-task-definition --region "${REGION}" --family "${TASK_DEFINTION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
    - echo "Updating the service..."
    - aws ecs update-service --region "${REGION}" --cluster "${CLUSTER_NAME}" --service "${SERVICE_NAME}"  --task-definition "${TASK_DEFINTION_NAME}"
  only:
    - master

update_task_definition_image.py

import sys, json, argparse

parser = argparse.ArgumentParser('Replaces image in the task definition')
parser.add_argument('image_uri', metavar='I', type=str, nargs='+',
                   help='The new image URI')

args = parser.parse_args()

definition = json.load(sys.stdin)['taskDefinition']['containerDefinitions']
definition[0]['image'] = args.image_uri[0]
print json.dumps(definition)

- docker build -t xxx .
- docker tag xxx:latest 733970770999.dkr.ecr.eu-west-2.amazonaws.com/xxx:latest
- echo "Pushing image..."
- docker push 733970770999.dkr.ecr.eu-west-2.amazonaws.com/nettools:latest
- $(aws ecr get-login --no-include-email --region eu-west-2| sed 's|https://||')
- aws ecs update-service --service xx-container-service --task-definition task-definition:20 --cluster xxx-cluster

Works fine!

Nice......

You can also add the following commands to check if the previous task was successfully stopped and the new one is started:

- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "Waiting until the previous task ${TASK_ID} is stopped..."
- aws ecs wait tasks-stopped --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"
- echo "The previous task ${TASK_ID} has stopped."
- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "New task ${TASK_ID} is running."
- aws ecs describe-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"

(y)

Hey, newbie question:

/bin/bash: line 102: apk: command not found

..isn't it support to be in the docker image?

Hey, newbie question:

/bin/bash: line 102: apk: command not found

..isn't it support to be in the docker image?

@guerzon APK is a package manager used by Alpine Linux (for example). It depends what image you're using. For Ubuntu, you might use APT.

Hey, newbie question:
/bin/bash: line 102: apk: command not found
..isn't it support to be in the docker image?

@guerzon APK is a package manager used by Alpine Linux (for example). It depends what image you're using. For Ubuntu, you might use APT.

Yeah I'm aware of that, which was why I was kicking myself when I found out I was using another image in the step where this error showed up. Thanks, all good now.

you can also use --query parameter instead of jq to get specific output:
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html#cli-usage-output-filter

Hey, newbie question:
/bin/bash: line 102: apk: command not found
..isn't it support to be in the docker image?

@guerzon APK is a package manager used by Alpine Linux (for example). It depends what image you're using. For Ubuntu, you might use APT.

Yeah I'm aware of that, which was why I was kicking myself when I found out I was using another image in the step where this error showed up. Thanks, all good now.

Hi @guerzon and @jlis
Can you share your gitlab-runner configuration?
I have the same problem as you.
Thanks

Hey, newbie question:
/bin/bash: line 102: apk: command not found
..isn't it support to be in the docker image?

@guerzon APK is a package manager used by Alpine Linux (for example). It depends what image you're using. For Ubuntu, you might use APT.

Yeah I'm aware of that, which was why I was kicking myself when I found out I was using another image in the step where this error showed up. Thanks, all good now.

Hi @guerzon and @jlis
Can you share your gitlab-runner configuration?
I have the same problem as you.
Thanks

Hi, here is my working configuration: https://gist.github.com/guerzon/3844df5399b28493136c940e0b5fbecd

Just to be clear, my issue was that the stage where I get this error was using another image, and it has nothing to do with my gitlab runner config. But hopefully the config works for you.

Hello,

I am running the same in GitlabCI but getting the authentication error in below command:

aws ecs register-task-definition --region "${AWS_DEFAULT_REGION}" --family "${TASK_DEFINITION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"

Error :"An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Private repository authentication requires that the task definition specify a value for executionRoleArn."

How to pass the credentials for private repo.

If you don't want to have external definition file you can also do this.

aws ecs register-task-definition --region ${region} --family ${task_defintion_name} --container-definitions " \
[
  {
    \"name\": \"hippo-ui\",
    \"image\": \"${image}\",
    \"cpu\": 10,
    \"memory\": 256,
    \"essential\": true,
    \"portMappings\": [
      {
        \"containerPort\": 4200
      }
    ],
    \"logConfiguration\": {
      \"logDriver\": \"awslogs\",
      \"options\": { 
        \"awslogs-group\" : \"/ecs/hippo-ui-container\",
        \"awslogs-region\": \"us-east-2\"
      }
    }
  }
]"

You can also add the following commands to check if the previous task was successfully stopped and the new one is started:

- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "Waiting until the previous task ${TASK_ID} is stopped..."
- aws ecs wait tasks-stopped --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"
- echo "The previous task ${TASK_ID} has stopped."
- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "New task ${TASK_ID} is running."
- aws ecs describe-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"

You can also add the following commands to check if the previous task was successfully stopped and the new one is started:

- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "Waiting until the previous task ${TASK_ID} is stopped..."
- aws ecs wait tasks-stopped --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"
- echo "The previous task ${TASK_ID} has stopped."
- TASK_ARN=$(aws ecs list-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --service-name "${SERVICE_NAME}")
- TASK_ID=$(echo ${TASK_ARN:72:36})
- echo "New task ${TASK_ID} is running."
- aws ecs describe-tasks --region "${REGION}" --cluster "${CLUSTER_NAME}" --tasks "${TASK_ID}"

If there are more than ONE task then it should be TASK_ID=$(echo ${TASK_ARN:82:32})

Anyone here has a solution to pass a container environment used on applications?

Like injecting this Envs in the task definition.

Note, in Gitlab you can now use $CI_COMMIT_SHORT_SHA instead of "$(echo $CI_COMMIT_SHA | head -c 8)". See: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html#:~:text=is%20built%20for.-,CI_COMMIT_SHORT_SHA,-11.7

Anyone here has a solution to pass a container environment used on applications?

Like injecting this Envs in the task definition.

I have used sed command in my pipeline to inject env vars in task definition file.

    - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$CI_AWS_ECS_TASK_DEFINITION" --region "${AWS_DEFAULT_REGION}")
    - NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$DOCKER_REGISTRY/$APP_NAME:latest" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')
    - echo "Registering new container definition..."
    - aws ecs register-task-definition --region "${AWS_DEFAULT_REGION}" --family "${CI_AWS_ECS_TASK_DEFINITION}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
    - echo "Updating the service..."
    - aws ecs update-service --region "${AWS_DEFAULT_REGION}" --cluster "${CI_AWS_ECS_CLUSTER}" --service "${CI_AWS_ECS_SERVICE}"  --task-definition "${CI_AWS_ECS_TASK_DEFINITION}"

this is the script i'm using however the service deployment does not start the desired task count is 1 however it does not create any

or you can use sed to set the new image tag, in my case, the CI_PIPELINE_IID and then use jq to extract the container definition:

NEW_CONTAINER_DEFINITION=$(echo $TASK_DEFINITION | sed -e "s/ecr-image:.*/ecr-image:$CI_PIPELINE_IID\",/g")
NEW_CONTAINER_DEFINITION=$(echo $NEW_CONTAINER_DEFINITION | jq '.taskDefinition.containerDefinitions[0]')

Actually there was nothing wrong with the script, it was failing to bind a port to the new container because I was using a static port binding.
Later I switched to using gitlab's ecs deployment script

    - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$CI_AWS_ECS_TASK_DEFINITION" --region "${AWS_DEFAULT_REGION}")
    - NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$DOCKER_REGISTRY/$APP_NAME:latest" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')
    - echo "Registering new container definition..."
    - aws ecs register-task-definition --region "${AWS_DEFAULT_REGION}" --family "${CI_AWS_ECS_TASK_DEFINITION}" --container-definitions "${NEW_CONTAINER_DEFINTIION}"
    - echo "Updating the service..."
    - aws ecs update-service --region "${AWS_DEFAULT_REGION}" --cluster "${CI_AWS_ECS_CLUSTER}" --service "${CI_AWS_ECS_SERVICE}"  --task-definition "${CI_AWS_ECS_TASK_DEFINITION}"

this is the script i'm using however the service deployment does not start the desired task count is 1 however it does not create any

you can set minimum healthy percent to 0 and maximum percent to 100 if you want only one container running on your ecs.

Getting an error while using was ecr login. Any idea? I'm passing secret key and access ID from CI/CD in variables

Hi Guys,

I have done with my gitlab pipeline one branh (master) and now i have fail with two branh both master, develop branh.

Please help me to wrire into .gitlab-ci.yml one
build:
stage: build
deploy:
stage: deploy

all develop, master branh?

Thank for help and hope see rely soon.

Hi there!
Just finished with ECS FARGATE deployment step at the GitLab CI. Want to share the result.
Hopefully, this would be helpful and save time for someone ;)

- TASK_DEFINITION_CURRENT=$(aws ecs describe-services --services "${ECS_SERVICE_NAME}" --cluster "${ECS_CLUSTER_NAME}" | jq -r .services[].taskDefinition | cut -d "/" -f 2)
- TASK_DEFINTION_NAME=$(echo $TASK_DEFINITION_CURRENT | cut -d ":" -f 1 )
- TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_DEFINTION_NAME")
- NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_SLUG}:${DEPLOY_CONTAINER_TAG}" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')
- EXECUTION_ROLE_ARN=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.executionRoleArn')
- TASK_ROLE_ARN=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.taskRoleArn')
- CONTAINER_MEMORY=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.memory')
- CONTAINER_CPU=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.cpu')
- REQUIRE_COMPATABILITIES=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.requiresCompatibilities[]')
- NETWORK_MODE=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.networkMode')
- echo "Registering new container definition..."
- aws ecs register-task-definition --family "${TASK_DEFINTION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}" --execution-role-arn "${EXECUTION_ROLE_ARN}" --memory "${CONTAINER_MEMORY}" --requires-compatibilities "${REQUIRE_COMPATABILITIES}" --network-mode "${NETWORK_MODE}" --cpu "${CONTAINER_CPU}"  --task-role-arn "${TASK_ROLE_ARN}"
- echo "Updating the service..."
- aws ecs update-service --cluster "${ECS_CLUSTER_NAME}" --service "${ECS_SERVICE_NAME}"  --task-definition "${TASK_DEFINTION_NAME}"

to avoid python you can use jq

- NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$REPOSITORY_URL:$IMAGE_TAG" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')

Hi,
Is there anyway we can pass two container images with this? I have been trying to work around this and could not find any solutions.

Hi Team ,

ia m getting below vunrability how to fix any one pls help

Improper access control in the Intel(R) Ethernet Controller RDMA driver for linux before version 1.9.30 may allow an unauthenticated user to potentially enable escalation of privilege via network access.

Hi Team ,

i am using for deployment below script

scripts:

• $(aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ECR_REGISTRY | tr -d '\r')
  • TASK_DEFINITION_CURRENT=$(aws ecs describe-services --services "${SERVICE_NAME}" --cluster "${CLUSTER_NAME}" | jq -r .services[].taskDefinition | cut -d "/" -f 2)
  • TASK_DEFINTION_NAME=$(echo $TASK_DEFINITION_CURRENT | cut -d ":" -f 1 )
  • TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_DEFINTION_NAME")
  • NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "${AWS_ECR_REGISTRY}/${APP_NAME}/${CI_COMMIT_REF_SLUG}:${SEMVER}" '.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions[0]')
  • EXECUTION_ROLE_ARN=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.executionRoleArn')
  • TASK_ROLE_ARN=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.taskRoleArn')
  • CONTAINER_MEMORY=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.memory')
  • CONTAINER_CPU=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.cpu')
  • REQUIRE_COMPATABILITIES=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.requiresCompatibilities[]')
  • NETWORK_MODE=$(echo $TASK_DEFINITION | jq -r '.taskDefinition.networkMode')
  • echo "Registering new container definition..."
  • aws ecs register-task-definition --family "${TASK_DEFINTION_NAME}" --container-definitions "${NEW_CONTAINER_DEFINTIION}" --execution-role-arn "${EXECUTION_ROLE_ARN}" --memory "${CONTAINER_MEMORY}" --requires-compatibilities "${REQUIRE_COMPATABILITIES}" --network-mode "${NETWORK_MODE}" --cpu "${CONTAINER_CPU}" --task-role-arn "${TASK_ROLE_ARN}"
  • echo "Updating the service..."
  • aws ecs update-service --cluster "${CLUSTER_NAME}" --service "${SERVICE_NAME}" --task-definition "${TASK_DEFINTION_NAME}"

but i am getting below error

“message”: “(service crm-api-secured) failed to launch a task with (error ECS was unable to assume the role ‘arn:aws:iam::XXXXXXXXX:role/null’ that was provided for this task. Please verify that the role being passed has the proper trust relationship and permissions and that your IAM user has permissions to pass this role.).”
any one pls help for the same.