Upgrade a Docker Secret in a Running Swarm Service

README.md

Update: I'm using this strategy, at the moment: https://github.com/jamiejackson/docker-secret-hash

---

This is a helper script for updating a secret in a running service. It does cause the service containers to restart (there's no avoiding that), but it does so as elegantly/conveniently as I know how.

See the motivation for this script, as well as alternatives, here.

Caveat:

• Secret names with a length approaching 64 characters might exhibit weird behavior, as this relies on a version token being tacked onto the secret name. Since the secret name is truncated to 64 characters, a long name might result in a smaller (or nonexistent) version token, leading to uniqueness problems and unexpected behavior.
• This won't update shared secret across multiple services. Maybe in the future, the stack option will be made optional and it will affect all relevant services. (Update: This script seems to address this, though I haven't tried it: https://gist.github.com/MLescaudron/e8248d32d3a5b8caaf622c1a829cf067 )

docker_secret_update.sh

#!/usr/bin/env bash
set -e

# ./docker_secret_update.sh --stack=hudx_local2 --service=lucee --secret=fr_admin_password --value=foobar

while :
do
  case $1 in
    --stack=*)
      stack=${1#*=}
      shift
      ;;
    --service=*)
      service=${1#*=}
      shift
      ;;
    --secret=*)
      secret=${1#*=}
      shift
      ;;
    --value=*)
      value=${1#*=}
      shift
      ;;
    --) # End of all options
      shift
      break
      ;;
    -*)
      echo "WARN: Unknown option (ignored): $1" >&2
      shift
      ;;
    *)  # no more options. Stop while loop
      break
      ;;
  esac
done

if [ -z "$stack" ] || [ -z "$service" ] || [ -z "$secret" ] || [ -z "$value" ]; then
    echo "Must provide --stack <stack_name> --service <service_name> --secret <secret_nane> --value <secret_value>" 1>&2
    exit 1
fi

# green echo
function myecho {
  BLACK=`tput setaf 0`
  RED=`tput setaf 1`
  GREEN=`tput setaf 2`
  YELLOW=`tput setaf 3`
  BLUE=`tput setaf 4`
  MAGENTA=`tput setaf 5`
  CYAN=`tput setaf 6`
  WHITE=`tput setaf 7`
  
  BOLD=`tput bold`
  RESET=`tput sgr0`
  
  echo -e "${GREEN}$1${RESET}"
}

secret_name=$secret
secret_value=$value

new_secret_version=_v-$(uuidgen)

stack_service=${stack}_${service}
stack_secret_name=${stack}_${secret_name}

stack_secret_name_old_version=$(docker service inspect --format "
  {{range .Spec.TaskTemplate.ContainerSpec.Secrets}}
    {{if eq .File.Name \"$secret_name\"}}
      {{.SecretName}}
    {{end}}
  {{end}}
  " "$stack_service" | tr -d '[:space:]' )

myecho "the existing service secret is named: $stack_secret_name_old_version"

# create new secret name; trim down to 64 characters (docker secret name limit)
stack_secret_name_new_version=$(  echo ${stack}_${secret_name}${new_secret_version} |  cut -c1-64 ) 

myecho "creating new secret: $stack_secret_name_new_version"
echo -n "$secret_value" | docker secret create "$stack_secret_name_new_version" - > /dev/null

myecho "remove old secret, add new, & bring up updated service"
docker service update \
  --secret-rm "$stack_secret_name_old_version" \
  --secret-add src="$stack_secret_name_new_version",target="$secret_name" \
  "$stack_service"

myecho "find old secret versions"
# todo: if we ever get into rolling back, will this cause problems?
secrets_to_remove=$(
  docker secret ls --filter name="${stack_secret_name}_v-" --format='{{.Name}}' -q \
   | grep -v "$stack_secret_name_new_version" \
   || true
)
secrets_to_remove="$secrets_to_remove $stack_secret_name"

myecho "removing secrets: $secrets_to_remove"
docker secret rm $secrets_to_remove || true

docker_secret_update_test.sh

#!/usr/bin/env bash

tearDown () {
  echo "= cleanup test remnants"
  rm -rf ./generated
  echo "= remove stack"
  docker stack rm my_stack || true
  echo "= remove secrets"
  docker secret rm $(docker secret ls -f name=my_stack_my_secret_name -q) || true
}

waitForStack () {
  stack=$1
  echo "= waiting for stack: $service_container"
  until $(docker stack ls | grep "^$stack\s" > /dev/null); do
    printf '.'
    sleep 1
  done
  echo ""
}

getServiceContainer () {
  stack_service=$1
  echo -n "$stack_service.1.$(docker service ps -f name=$stack_service.1 $stack_service -q --no-trunc | head -n1)"
}

execInService () {
  stack_service=$1
  cmd=$2
  service_container=$(getServiceContainer "$stack_service")
  docker exec -ti "$service_container" $cmd
}

waitForServiceContainer () {
  service=$1
  container=$(getServiceContainer "$service")
  echo "= waiting for container: $container"
  until $(docker ps --format '{{.Names}}' | grep "$container" > /dev/null); do
    printf '.'
    sleep 1
  done
  echo ""
}

getSecretValueInService () {
  service=$1
  secret_name=$2
  waitForServiceContainer "$service" > /dev/null
  echo -n $(execInService $service "cat /run/secrets/$secret_name")
}

tearDown

echo "= setup test"
mkdir ./generated
chmod a+x ./docker_secret_update.sh

echo "= create secret file"
echo -n "old_secret_value" > ./generated/my_secret_file

echo "= generate docker-compose.yml"
cat > ./generated/docker-compose.yml <<EOF
---
version: '3.6'
services:

  my_service:
    image: nginx
    secrets:
      - my_secret_name
#  my_service_2:
#    image: nginx
#    secrets:
#      - my_secret_name
secrets:
  my_secret_name:
    file: ./my_secret_file
EOF

echo "= deploy stack"
docker stack deploy -c ./generated/docker-compose.yml my_stack

echo "= waiting for stack"
waitForStack "my_stack"
echo "= waiting for container"
waitForServiceContainer "my_stack_my_service"

echo "= secret's value in service is: $(getSecretValueInService my_stack_my_service my_secret_name)"

echo "= update secret file value"
echo -n "new secret value" > ./generated/my_secret_file

echo "= (known to be failing) attempt at secret update"
docker stack deploy -c ./generated/docker-compose.yml my_stack || true

echo "= trying JJ's solution"
./docker_secret_update.sh --stack=my_stack --service=my_service \
  --secret=my_secret_name --value="new_secret_value"

echo "= secret's value in service is: $(getSecretValueInService my_stack_my_service my_secret_name)"

echo "= (successful) attempt at secret update"
docker stack deploy -c ./generated/docker-compose.yml my_stack

echo "= secret's value in service is: $(getSecretValueInService my_stack_my_service my_secret_name)"

tearDown

Hi, thanks for this. But I can't figure out how to make the comparison (line 71 in docker_secret_update) go with the beginning of the config name. Example, my File.Name is " conf_name_1", and my secret_name is "conf_na", how if eq File.Name is startwith "conf_na"?