-
-
Save janjaapbos/b67f97f2f32d7cf09c066fa5eaf50e89 to your computer and use it in GitHub Desktop.
version: '2.1' | |
# run with IPv6 network of the docker container as enviroment variable | |
# e.g. ZT6PLANE=fc7b:59ab:4811:901c:40ea docker-compose up | |
networks: | |
zerotier: | |
driver: bridge | |
enable_ipv6: true | |
internal: false | |
ipam: | |
config: | |
- subnet: ${ZT6PLANE}::/80 | |
volumes: | |
zerotier_var: | |
services: | |
zerotier: | |
image: zerotier/zerotier-containerized | |
devices: | |
- /dev/net/tun | |
network_mode: host | |
cap_add: | |
- NET_ADMIN | |
- SYS_ADMIN | |
volumes: | |
- zerotier_var:/var/lib/zerotier-one/ | |
# this only exists so that the networks get created | |
alpine: | |
image: bwstitt/alpine | |
command: tail -f /dev/null | |
# uncomment this once the zerotier container is running | |
networks: | |
zerotier: | |
ipv6_address: ${ZT6PLANE}::2 |
On the server try a ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
If that works, do a tcpdump on on eth0 in the fcf0:a9af:17a3:c742:eb37::0c4d:421f container and do on your mac a ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f, and check what you see with tcpdump.
On my server to a container on my server:
$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING fcf0:a9af:17a3:c742:eb37::0c4d:421f(fcf0:a9af:17a3:c742:eb37:0:c4d:421f) 56 data bytes
From fcf0:a9af:17a3:c742:eb37::1 icmp_seq=1 Destination unreachable: Address unreachable
--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping statistics ---
4 packets transmitted, 0 received, +1 errors, 100% packet loss, time 3082ms
$ traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37:0:c4d:421f), 30 hops max, 80 byte packets
1 tank.stitthappens.com (fcf0:a9af:17a3:c742:eb37::1) 3054.881 ms !H 3054.808 ms !H 3054.796 ms !H
On my laptop:
$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17a3:c742:eb37::c4d:421f
--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
$ traceroute6 fcf0:a9af:17a3:c742:eb37::c4d:421f
traceroute6 to fcf0:a9af:17a3:c742:eb37::c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
1 fcf0:a9af:17a3:c742:eb37::1 4.301 ms 10.904 ms 2.631 ms
2 fcf0:a9af:17a3:c742:eb37::1 3062.897 ms !A 3081.672 ms !A 3126.021 ms !A
Here is the tcpdump while that ping/traceroute was happening:
/ # tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:04:50.764308 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:04:50.764386 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:20.764321 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:05:20.764456 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:35.135793 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:36.138648 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:37.162657 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:40.216394 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:41.258636 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:42.282634 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:50.764303 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:05:50.764736 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:50.794645 IP6 fe80::42:acff:fe12:6 > ip6-allrouters: ICMP6, router solicitation, length 16
21:06:09.346567 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:10.378639 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:11.402634 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:12.429647 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:13.450630 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:14.474653 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:15.503637 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:16.522636 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:17.546658 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:20.764303 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:06:20.764465 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
What is really odd is ping/traceroute works from my laptop to my server for a different container:
$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::2
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=2 ttl=64 time=0.150 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=3 ttl=64 time=0.159 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=4 ttl=64 time=0.150 ms
--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.063/0.130/0.159/0.040 ms
$ traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute6 to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
1 fcf0:a9af:17a3:c742:eb37::1 3.607 ms 2.281 ms 1.627 ms
2 fcf0:a9af:17a3:c742:eb37::2 1.381 ms 1.096 ms 1.345 ms
Ok, so if on the server it already does not work to ping its container, you can focus on getting that to work before testing across hosts. So the tcpdump is done on the container where the ping is directed? You see NDP request but no responses.
Can you ping between containers on the same host?
What is the host OS / distro?
Is there a host firewall active?
Yes, the tcpdump was done inside my haproxy container from this command: docker run --rm -it --net container:frontend_haproxy_zt_1 nicolaka/netshoot
I am able to ping between some containers on the same host (::b37e:f2a9 -> ::2):
# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 -c2 fcf0:a9af:17a3:c742:eb37::2
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=2 ttl=64 time=0.085 ms
--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.085/0.138/0.192/0.054 ms
# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
1 shared_alpine_zt_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::2) 0.011 ms 0.005 ms 0.002 ms
It fails for this other host though (::b37e:f2a9 -> ::c4d:421f):
# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f), 30 hops max, 72 byte packets
1 ethereum_parity_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::b37e:f2a9) 3075.682 ms !H 3071.421 ms !H 3071.836 ms !H
# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING fcf0:a9af:17a3:c742:eb37::0c4d:421f(fcf0:a9af:17a3:c742:eb37::c4d:421f) 56 data bytes
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=1 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=5 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=6 Destination unreachable: Address unreachable
^C
--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping statistics ---
8 packets transmitted, 0 received, +3 errors, 100% packet loss, time 7148ms
Host OS is Fedora 26 with the latest ce version of docker (installed via docker-machine). I've disabled the firewall to simplify the testing.
Regarding NDP on the host, perhaps this helps:
sysctl -w net.ipv6.conf.all.proxy_ndp=1
Looks promising!
[bwstitt@tank:~] $ sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0
[admin@aws:~] $ sudo sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0
But changing it to 1 doesn't seem to have made any difference. Pings still fail with the same errors
Latest tcpdump output
[bwstitt:~] $ docker run -it --net host nicolaka/netshoot sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1
[bwstitt:~] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::4
traceroute to fcf0:a9af:17a3:c742:eb37::4 (fcf0:a9af:17a3:c742:eb37::4), 30 hops max, 72 byte packets
1 fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1) 299.604 ms 312.555 ms 338.743 ms
2 * * *
3 * * *
4 * * *
5 * * *^C
[root@tank] # docker run -it --net host nicolaka/netshoot sysctl nev6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1
[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:05:36.083679 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33435: UDP, length 24
02:05:36.083873 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:36.327194 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33436: UDP, length 24
02:05:36.327332 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:39.686508 IP 10.242.176.103.54421 > 10.242.255.255.21027: UDP, length 69
02:05:41.450627 IP6 fe80::4ca2:c1ff:fe21:b299 > fcf0:a9af:17ea:c412:57de::1: ICMP6, neighbor solicitation, who has fcf0:a9af:17ea:c412:57de::1, length 32
02:05:41.562750 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33438: UDP, length 24
02:05:41.756616 IP6 fcf0:a9af:17ea:c412:57de::1 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fcf0:a9af:17ea:c412:57de::1, length 24
02:05:46.577343 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33439: UDP, length 24
02:05:46.634252 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:46.634393 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:46.790118 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor solicitation, who has fe80::4ca2:c1ff:fe21:b299, length 32
02:05:46.790142 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor advertisement, tgt is fe80::4ca2:c1ff:fe21:b299, length 24
02:05:47.710453 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:47.710635 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:48.733475 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:48.733608 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:51.690628 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor solicitation, who has fe80::4ceb:c2ff:fe71:e70, length 32
02:05:51.909489 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fe80::4ceb:c2ff:fe71:e70, length 24
And here is a successful traceroute for a different container on the same host:
[bwstitt@laptop] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
1 fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1) 245.199 ms * 324.855 ms
2 fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2) 305.673 ms 309.446 ms 309.176 ms
[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:13:54.875535 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33435: UDP, length 24
02:13:54.875678 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.145734 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33437: UDP, length 24
02:14:00.145908 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.459565 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33438: UDP, length 24
02:14:00.459606 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33438, length 80
02:14:00.763468 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33439: UDP, length 24
02:14:00.763519 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33439, length 80
02:14:01.081223 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33440: UDP, length 24
02:14:01.081259 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33440, length 80
Looks like zerotier/zerotier-containerized
is gone :(
Still stumped.
fcf0:a9af:17ea:c412:57de::1 is my docker-for-mac VM
fcf0:a9af:17a3:c742:eb37::/80 is my server
On my laptop inside the docker-for-mac VM:
On my server: