Docker EE Inline upgrade procedure

Inline_Upgrade.md

Docker EE Inline upgrade procedure

This is a guide that details the steps to have in place a duplicate environment to upgrade to, and requires an existing UCP/DTR cluster.

Prerequisites

Ensure that new nodes are all confiured with identical firewall rules and that all of the relevant swarm joins are performed.

• Create a backup from existing UCP Cluster (UCP Will stop during the backup)

Ensure you use the correct tag to see what is running, check in docker images

docker container run --log-driver none --rm -i --name ucp \
-v /var/run/docker.sock:/var/run/docker.sock \
docker/ucp:2.1.5 backup --interactive > /tmp/backup.tar


INFO[0000] Your engine version 17.03.2-ee-6, build bdc2646 (3.10.0-514.26.2.el7.x86_64) is compatible 
INFO[0000] We're about to temporarily stop all local UCP containers for UCP ID: TV5U:YMWJ:TVUL:TMZ5:AGGM:XBWC:7X5E:TSUL:TAVN:K6GD:DBUQ:XMSP 
Do you want proceed with the backup? (y/n): y
INFO[0023] Temporarily stopping local UCP containers to ensure a consistent backup 
INFO[0085] Backing up internal KV store                 
INFO[0000] Beginning backup                             
INFO[0003] Backup completed successfully                
INFO[0090] Resuming stopped UCP containers     

• Copy the backup to a new UCP node

• Create a DTR backup

Locate the DTR Replica ID by looking in the web UI (Resources->Stacks & Applications) and drilling down to look at the host.

$ export UCP_PASSWORD=<>
$ docker run --log-driver none -i --rm   --env UCP_PASSWORD=$UCP_PASSWORD \   
docker/dtr:2.2.8 backup   --ucp-url http://ucp.dckr.org   --ucp-insecure-tls \   
--ucp-username admin   --existing-replica-id <REPLICA_ID> > backup-metadata.tar

INFO[0000] Validating UCP cert                          
INFO[0000] Connecting to UCP                            
INFO[0000] UCP cert validation successful               
INFO[0000] Validating UCP cert                          
INFO[0000] Connecting to UCP                            
INFO[0000] UCP cert validation successful               
INFO[0000] Connecting to network: dtr-ol                
INFO[0000] Waiting for phase2 container to be known to the Docker daemon 
INFO[0011] Backup complete.      

• Copy the backup to a new DTR node

• Restore that backup on a new node (which will be part of the new cluster)

This will create a new swarm cluster, that we will add in additional managers.

docker container run --rm -i --name ucp \
   -v /var/run/docker.sock:/var/run/docker.sock  \
   docker/ucp:2.1.5 restore --host-address <IP_ADDRESS> --san <UCP_LB> --san <DTR_LB> < ./backup.tar 

Unable to find image 'docker/ucp:2.1.5' locally
2.1.5: Pulling from docker/ucp
90f4dba627d6: Pull complete 
e02efbcff02d: Pull complete 
6cd12384bafd: Pull complete 
Digest: sha256:2ccca84731fb8e0050f85d009769250c4e0d0d4e8c7ff85a229cef90b70ee8d0
Status: Downloaded newer image for docker/ucp:2.1.5
time="2017-09-13T13:58:08Z" level=info msg="Your engine version 17.03.2-ee-6, build bdc2646 (3.10.0-514.26.2.el7.x86_64) is compatible" 
time="2017-09-13T13:58:08Z" level=info msg="All required images are present" 
time="2017-09-13T13:58:08Z" level=warning msg="Restore is running in non-interactive mode. Proceeding to read backup file from stdin" 
time="2017-09-13T13:58:09Z" level=info msg="Determining Swarm state" 
time="2017-09-13T13:58:09Z" level=info msg="Parsing backup file" 
time="2017-09-13T13:58:09Z" level=info msg="This node is not participating in a swarm, creating a new swarm" 
time="2017-09-13T13:58:10Z" level=info msg="Initializing a new swarm at 172.31.29.168" 
time="2017-09-13T13:58:10Z" level=info msg="Swarm successfully initialized" 
time="2017-09-13T13:58:17Z" level=info msg="Deploying UCP Agent Service" 
...

• Add additional cluster managers Get the join tokens from the restored UCP UI

docker swarm join --token <TKN> x.x.x.x:2377

• Modify the repository from 17.03 -> 17.06

sudo vi /etc/yum.repos.d/docker-ee.repo and change the .03 references to .06 and then rebuild the cache with sudo yum makecache fast. Then confirm the new packages with:

sudo yum list docker-ee.x86_64  --showduplicates | sort -r
<...>
docker-ee.x86_64        17.06.1.ee.2-1.el7.centos        docker-ee-stable-17.06 
docker-ee.x86_64        17.06.1.ee.1-1.el7.centos        docker-ee-stable-17.06 
docker-ee.x86_64        17.03.2.ee.6-1.el7.centos        @docker-ee-stable-17.03

• Upgrade Docker Engine

Once the repository has been confirmed, the engine can be upgraded with the following:

$ sudo yum -y install docker-ee

• Upgrade UCP

$ docker container run --rm -it \
   --name ucp \
   -v /var/run/docker.sock:/var/run/docker.sock \
   docker/ucp:2.2.2 \
   upgrade --interactive

• Remove original UCP instances from load balancer

This is required as DTR uses the UCP load balancer address when configuring the SSO etc.. so having the original UCP nodes will break/confuse things.

• Upgrade DTR node to Docker 17.06 as before

• Add DTR node to swarm cluster

• Remove previous DTR nodes from load balancer (leaving only the new DTR)

• Restore DTR from backup

docker run -i --rm \
  docker/dtr:2.2.8 restore \
  --ucp-url https://ucp.dckr.org \
  --ucp-insecure-tls \
  --ucp-username admin \
  --ucp-node dtr4.dckr.org \
  --replica-id e227cbdfac45 \
  --dtr-external-url https://dtr.dckr.org < /tmp/backup-metadata.tar

• Test DTR

Log into the DTR through the web UI and ensure that all settings are migrated correctly, S3 backend etc..

• Upgrade DTR

docker run -it --rm \
   docker/dtr:2.3.2 upgrade \
   --ucp-insecure-tls \
   --ucp-url https://ucp.dckr.org \
   --ucp-username admin \
   --ucp-password <PW>

• Add Replicas

• Fill load balancer settings with the additional Managers and DTR instances

Something went wrong!

Restore load balancer entries with original addresses of old UCP and DTR

Great! Thanks, Dan! I'll update my overall procedure with your findings & approach.