docker compose for ZeroTier 6plane

docker-compose.yml

version: '2.1'

# run with IPv6 network of the docker container as enviroment variable
# e.g. ZT6PLANE=fc7b:59ab:4811:901c:40ea docker-compose up

networks:
  zerotier:
    driver: bridge
    enable_ipv6: true
    internal: false
    ipam:
      config:
        - subnet: ${ZT6PLANE}::/80

volumes:
  zerotier_var:

services:
  zerotier:
    image: zerotier/zerotier-containerized
    devices:
      - /dev/net/tun
    network_mode: host
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
    volumes:
      - zerotier_var:/var/lib/zerotier-one/

  # this only exists so that the networks get created
  alpine:
    image: bwstitt/alpine
    command: tail -f /dev/null
    # uncomment this once the zerotier container is running
    networks:
      zerotier:
       ipv6_address: ${ZT6PLANE}::2

Yes, the tcpdump was done inside my haproxy container from this command: docker run --rm -it --net container:frontend_haproxy_zt_1 nicolaka/netshoot

I am able to ping between some containers on the same host (::b37e:f2a9 -> ::2):

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 -c2 fcf0:a9af:17a3:c742:eb37::2
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=2 ttl=64 time=0.085 ms

--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.085/0.138/0.192/0.054 ms


# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
 1  shared_alpine_zt_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::2)  0.011 ms  0.005 ms  0.002 ms

It fails for this other host though (::b37e:f2a9 -> ::c4d:421f):

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f), 30 hops max, 72 byte packets
 1  ethereum_parity_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::b37e:f2a9)  3075.682 ms !H  3071.421 ms !H  3071.836 ms !H

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING fcf0:a9af:17a3:c742:eb37::0c4d:421f(fcf0:a9af:17a3:c742:eb37::c4d:421f) 56 data bytes
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=1 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=5 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=6 Destination unreachable: Address unreachable
^C
--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping statistics ---
8 packets transmitted, 0 received, +3 errors, 100% packet loss, time 7148ms

Host OS is Fedora 26 with the latest ce version of docker (installed via docker-machine). I've disabled the firewall to simplify the testing.

Regarding NDP on the host, perhaps this helps:
sysctl -w net.ipv6.conf.all.proxy_ndp=1

Looks promising!

[bwstitt@tank:~] $ sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0

[admin@aws:~] $ sudo sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0

But changing it to 1 doesn't seem to have made any difference. Pings still fail with the same errors

Latest tcpdump output

[bwstitt:~] $ docker run -it --net host nicolaka/netshoot sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1

[bwstitt:~] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::4
traceroute to fcf0:a9af:17a3:c742:eb37::4 (fcf0:a9af:17a3:c742:eb37::4), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  299.604 ms  312.555 ms  338.743 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *^C

[root@tank] # docker run -it --net host nicolaka/netshoot sysctl nev6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1

[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:05:36.083679 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33435: UDP, length 24
02:05:36.083873 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:36.327194 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33436: UDP, length 24
02:05:36.327332 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:39.686508 IP 10.242.176.103.54421 > 10.242.255.255.21027: UDP, length 69
02:05:41.450627 IP6 fe80::4ca2:c1ff:fe21:b299 > fcf0:a9af:17ea:c412:57de::1: ICMP6, neighbor solicitation, who has fcf0:a9af:17ea:c412:57de::1, length 32
02:05:41.562750 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33438: UDP, length 24
02:05:41.756616 IP6 fcf0:a9af:17ea:c412:57de::1 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fcf0:a9af:17ea:c412:57de::1, length 24
02:05:46.577343 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33439: UDP, length 24
02:05:46.634252 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:46.634393 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:46.790118 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor solicitation, who has fe80::4ca2:c1ff:fe21:b299, length 32
02:05:46.790142 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor advertisement, tgt is fe80::4ca2:c1ff:fe21:b299, length 24
02:05:47.710453 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:47.710635 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:48.733475 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:48.733608 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:51.690628 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor solicitation, who has fe80::4ceb:c2ff:fe71:e70, length 32
02:05:51.909489 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fe80::4ceb:c2ff:fe71:e70, length 24

And here is a successful traceroute for a different container on the same host:

[bwstitt@laptop] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  245.199 ms  *  324.855 ms
 2  fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2)  305.673 ms  309.446 ms  309.176 ms

[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:13:54.875535 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33435: UDP, length 24
02:13:54.875678 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.145734 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33437: UDP, length 24
02:14:00.145908 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.459565 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33438: UDP, length 24
02:14:00.459606 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33438, length 80
02:14:00.763468 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33439: UDP, length 24
02:14:00.763519 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33439, length 80
02:14:01.081223 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33440: UDP, length 24
02:14:01.081259 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33440, length 80

Looks like zerotier/zerotier-containerized is gone :(