Skip to content

Instantly share code, notes, and snippets.

@tsabat

tsabat/DOCKER_CONSTRAINT_GLOBAL_TEST.md

Last active October 24, 2016 21:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tsabat/de43e3384b5bbc0be2f7a13009785c82 to your computer and use it in GitHub Desktop.
Save tsabat/de43e3384b5bbc0be2f7a13009785c82 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
DOCKER_CONSTRAINT_GLOBAL_TEST.md

We're testing what happens when you try to update the the image on a global service, placed with constraints.

Prerequisites

  • Install Docker 1.12.2 on ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20160907.1 (ami-746aba14)
  • Add Labels
echo '[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --userns-remap=default --storage-driver aufs --label codepen.service="cpor"' > /etc/systemd/system/docker.service.d/namespaces.conf
systemctl daemon-reload
systemctl restart docker.service

Create Swarm

on manager

docker swarm init

on workers

docker swarm join \
    --token SWMTKN-1-62j8irobsukkme5ug8qfrmqqmvzxb8dnrnxo3fx3i83gis55oh-80zkoquamsohk29h8s2vz41s8 \
    10.0.60.244:2377

check state on manager

ubuntu@ip-10-0-60-244:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/231072.231072/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: active
 NodeID: 2oxnez8pyggziugmlpwv0r22p
 Is Manager: true
 ClusterID: 6xkf30pvzaneeoe6tqpzqsi5c
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.0.60.244
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-36-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.795 GiB
Name: ip-10-0-60-244
ID: 7F5H:EHRO:LR6F:ECNY:6WSC:EMDL:6NLL:4TKO:ROWW:KK4Q:TR7Y:UQQZ
Docker Root Dir: /var/lib/docker/231072.231072
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 codepen.service=cpor
Insecure Registries:
 127.0.0.0/8

launch two nodes, label them, and join them to the manager

node a

echo /etc/systemd/system/docker.service.d/namespaces.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --userns-remap=default --storage-driver aufs --label codepen.service="node_a"
sudo systemctl daemon-reload && sudo systemctl restart docker.service
docker swarm join \
     --token SWMTKN-1-62j8irobsukkme5ug8qfrmqqmvzxb8dnrnxo3fx3i83gis55oh-80zkoquamsohk29h8s2vz41s8 \
     10.0.60.244:2377
This node joined a swarm as a worker.

node b

echo /etc/systemd/system/docker.service.d/namespaces.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --userns-remap=default --storage-driver aufs --label codepen.service="node_b"
sudo systemctl daemon-reload && sudo systemctl restart docker.service
docker swarm join \
     --token SWMTKN-1-62j8irobsukkme5ug8qfrmqqmvzxb8dnrnxo3fx3i83gis55oh-80zkoquamsohk29h8s2vz41s8 \
     10.0.60.244:2377
This node joined a swarm as a worker.

meanwhile, back at the manager

docker node update --availability drain 2ox
2ox
docker node ls
ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
2oxnez8pyggziugmlpwv0r22p *  ip-10-0-60-244  Ready   Drain         Leader
4nlayc2uzdal7wm9s6r0fjsgg    ip-10-0-61-82   Ready   Active
6g7d4k58lxc3njj3rd9u9lqbu    ip-10-0-62-74   Ready   Active

Create Services

service 1

docker service create --constraint 'engine.labels.codepen.service == node_a' --mode global --name redis redis:2.6.17
2809ondmn4m1exmhh4k49uwne

docker service ls
ID            NAME   REPLICAS  IMAGE         COMMAND
2809ondmn4m1  redis  global    redis:2.6.17

docker service ps redis
ID                         NAME       IMAGE         NODE           DESIRED STATE  CURRENT STATE                 ERROR
1002fvr5d1f6gi1p6tvgwuans  redis      redis:2.6.17  ip-10-0-62-74  Running        Allocated about a minute ago
1tjfsloodt61z1y1qgpvtnska   \_ redis  redis:2.6.17  ip-10-0-61-82  Running        Running about a minute ago

service 2

docker service create --constraint 'engine.labels.codepen.service == node_b' --mode global --name redis2 redis:3.2.4
bdipxy0oq9vtzgbp2vnekdjf3

docker service ls
ID            NAME    REPLICAS  IMAGE         COMMAND
2809ondmn4m1  redis   global    redis:2.6.17
bdipxy0oq9vt  redis2  global    redis:3.2.4

docker service ps redis2
ID                         NAME        IMAGE        NODE           DESIRED STATE  CURRENT STATE             ERROR
0jal8lx11fm1muiu640xcjpbb  redis2      redis:3.2.4  ip-10-0-61-82  Running        Allocated 54 seconds ago
3r90m628gqp9hfkuhx5kebyfh   \_ redis2  redis:3.2.4  ip-10-0-62-74  Running        Running 43 seconds ago

Here's where it gets interesting. The service redis2 which has "Running/Allocated" as the top entry in docker service ps and the "Running/Running" states below. So, if we try to update that service to a new image, it sets the state to Allocated on the top node, but none of the nodes below.

ubuntu@ip-10-0-60-244:~$ docker service update --image redis:2.6.17 redis2
redis2
ubuntu@ip-10-0-60-244:~$ docker service ps redis2
ID                         NAME        IMAGE         NODE           DESIRED STATE  CURRENT STATE            ERROR
bjkt1hh8fcrkrjqugyfhil6jr  redis2      redis:2.6.17  ip-10-0-61-82  Ready          Allocated 6 seconds ago
0jal8lx11fm1muiu640xcjpbb   \_ redis2  redis:3.2.4   ip-10-0-61-82  Shutdown       Allocated 3 minutes ago
3r90m628gqp9hfkuhx5kebyfh   \_ redis2  redis:3.2.4   ip-10-0-62-74  Running        Running 3 minutes ago
ubuntu@ip-10-0-60-244:~$ docker service ps redis2
ID                         NAME        IMAGE         NODE           DESIRED STATE  CURRENT STATE             ERROR
bjkt1hh8fcrkrjqugyfhil6jr  redis2      redis:2.6.17  ip-10-0-61-82  Ready          Allocated 17 seconds ago
0jal8lx11fm1muiu640xcjpbb   \_ redis2  redis:3.2.4   ip-10-0-61-82  Shutdown       Allocated 3 minutes ago
3r90m628gqp9hfkuhx5kebyfh   \_ redis2  redis:3.2.4   ip-10-0-62-74  Running        Running 3 minutes ago
ubuntu@ip-10-0-60-244:~$ docker service ps redis2
ID                         NAME        IMAGE         NODE           DESIRED STATE  CURRENT STATE             ERROR
bjkt1hh8fcrkrjqugyfhil6jr  redis2      redis:2.6.17  ip-10-0-61-82  Ready          Allocated 24 seconds ago
0jal8lx11fm1muiu640xcjpbb   \_ redis2  redis:3.2.4   ip-10-0-61-82  Shutdown       Allocated 3 minutes ago
3r90m628gqp9hfkuhx5kebyfh   \_ redis2  redis:3.2.4   ip-10-0-62-74  Running        Running 3 minutes ago
ubuntu@ip-10-0-60-244:~$ docker service ps redis2
ID                         NAME        IMAGE         NODE           DESIRED STATE  CURRENT STATE             ERROR
bjkt1hh8fcrkrjqugyfhil6jr  redis2      redis:2.6.17  ip-10-0-61-82  Ready          Allocated 44 seconds ago
0jal8lx11fm1muiu640xcjpbb   \_ redis2  redis:3.2.4   ip-10-0-61-82  Shutdown       Allocated 3 minutes ago
3r90m628gqp9hfkuhx5kebyfh   \_ redis2  redis:3.2.4   ip-10-0-62-74  Running        Running 3 minutes ago

This image depicts what is happening:

example

What's even more misleading, is that if the node is top in the list and Allocated/Running and you update the service, it will update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment