Skip to content

Instantly share code, notes, and snippets.

@janjaapbos

janjaapbos/docker-compose.yml

Created October 3, 2017 11:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save janjaapbos/b67f97f2f32d7cf09c066fa5eaf50e89 to your computer and use it in GitHub Desktop.
Save janjaapbos/b67f97f2f32d7cf09c066fa5eaf50e89 to your computer and use it in GitHub Desktop.
Download ZIP
docker compose for ZeroTier 6plane
Raw
docker-compose.yml
version: '2.1'
# run with IPv6 network of the docker container as enviroment variable
# e.g. ZT6PLANE=fc7b:59ab:4811:901c:40ea docker-compose up
networks:
zerotier:
driver: bridge
enable_ipv6: true
internal: false
ipam:
config:
- subnet: ${ZT6PLANE}::/80
volumes:
zerotier_var:
services:
zerotier:
image: zerotier/zerotier-containerized
devices:
- /dev/net/tun
network_mode: host
cap_add:
- NET_ADMIN
- SYS_ADMIN
volumes:
- zerotier_var:/var/lib/zerotier-one/
# this only exists so that the networks get created
alpine:
image: bwstitt/alpine
command: tail -f /dev/null
# uncomment this once the zerotier container is running
networks:
zerotier:
ipv6_address: ${ZT6PLANE}::2
@janjaapbos
Copy link
Author

Based on https://gist.github.com/WyseNynja/5ca0b962f6643b1459c6b410347ff10e

Works across containers on different docker hosts (mac / linux). Each docker host must define the ZT6PLANE environment variable based on the 6plane address of its ZeroTier container.

Note that the ipv6_address specification of the alpine container is optional. It will be auto assigned if not specified.

@BlinkyStitt
Copy link

BlinkyStitt commented Oct 3, 2017 •
edited

Using an environment variable is really clean. Thanks for that idea.

Unfortunately, I'm still having an issue with this. All hosts can ping all other hosts at $ZT6PLANE::1, but only my docker-for-mac VM has a functioning ::2.

Would tcpdump/ping/traceroute/etc be useful? If so, from where to where and with what flags?

fcf0:a9af:17a3:c742:eb37::/80 is my personal server.

fcf0:a9af:17ea:c412:57de::/80 is the docker-for-mac VM.

[admin@aws:~] $ ping6 -c1 fcf0:a9af:17a3:c742:eb37::1    # works
PING fcf0:a9af:17a3:c742:eb37::1(fcf0:a9af:17a3:c742:eb37::1) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::1: icmp_seq=1 ttl=64 time=82.3 ms

--- fcf0:a9af:17a3:c742:eb37::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 82.372/82.372/82.372/0.000 ms

[admin@aws:~] $ ping6 -c1 fcf0:a9af:17a3:c742:eb37::2    # fails
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes

--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

[admin@aws:~] 10s 1 $ ping6 -c1 fcf0:a9af:17ea:c412:57de::1    # works
PING fcf0:a9af:17ea:c412:57de::1(fcf0:a9af:17ea:c412:57de::1) 56 data bytes
64 bytes from fcf0:a9af:17ea:c412:57de::1: icmp_seq=1 ttl=64 time=103 ms

--- fcf0:a9af:17ea:c412:57de::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 103.733/103.733/103.733/0.000 ms

[admin@aws:~] $ ping6 -c1 fcf0:a9af:17ea:c412:57de::2    # works!
PING fcf0:a9af:17ea:c412:57de::2(fcf0:a9af:17ea:c412:57de::2) 56 data bytes
64 bytes from fcf0:a9af:17ea:c412:57de::2: icmp_seq=1 ttl=63 time=132 ms

--- fcf0:a9af:17ea:c412:57de::2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 132.544/132.544/132.544/0.000 ms

@janjaapbos
Copy link
Author

Perhaps check ipv6 forwarding parameter on linux hosts? Should be 1.

cat /proc/sys/net/ipv6/conf/all/forwarding
1

@janjaapbos
Copy link
Author

@wysenynja You can also check the forwarding on linux host for the specific br-xxx bridge that is auto configured by docker.

Can verify again that within the linux host the alpine container can be pinged at ::2?

@BlinkyStitt
Copy link

On all hosts:

$ cat /proc/sys/net/ipv6/conf/all/forwarding
1

This is extra weird. Figuring that we would eventually sort this out, I started adding addresses to the rest of my containers on my server and now I can ping the ::2 on my server from any host. What's weird is that I still can only ping the ::2 on my aws host from my aws host. None of the addresses I added to my other containers work. At least now I have 2 working ::2 addresses.

@BlinkyStitt
Copy link

BlinkyStitt commented Oct 3, 2017 •
edited

Maybe this is related?

inside a shared_alpine container on my laptop with only the zerotier network where pings work:

/ # ip -f inet6 route
fcf0:a9af:17a3:c742:eb37::/80 dev eth0  metric 256 
fe80::/64 dev eth0  metric 256 
default via fcf0:a9af:17a3:c742:eb37::1 dev eth0  metric 1024 
ff00::/8 dev eth0  metric 256 


/ # ip -f inet6 address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3180: eth0@if3181: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP 
    inet6 fcf0:a9af:17a3:c742:eb37::2/80 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:2/64 scope link 
       valid_lft forever preferred_lft forever

inside a shared_alpine container on my aws host with only the zerotier network where pings do not work:

/ # ip -f inet6 route
fcf0:a9af:177d:ae37:799d::/80 dev eth0  metric 256 
fe80::/64 dev eth0  metric 256 
default via fcf0:a9af:177d:ae37:799d::1 dev eth0  metric 1024 
unreachable default dev lo  metric -1  error -101
ff00::/8 dev eth0  metric 256 
unreachable default dev lo  metric -1  error -101


/ # ip -f inet6 address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
252: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 
    inet6 fcf0:a9af:177d:ae37:799d::2/80 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe14:2/64 scope link 
       valid_lft forever preferred_lft forever

Docker did the networking setup, so I'm not sure why they are different. Any ideas how to fix the aws host?

@BlinkyStitt
Copy link

Looks like docker isn't setting up the gateway. I'm going to try specifying it manually:

[admin@aws:~] $ sudo docker network inspect shared_zerotier
[
    {
        "Name": "shared_zerotier",
        "Id": "78aeae7b9a107dc62f8edeabb42643be462f78d518f85170daadc29a760e7e4c",
        "Created": "2017-10-03T20:41:44.963687708Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                },
                {
                    "Subnet": "fcf0:a9af:177d:ae37:799d::/80"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "760bdcec97146204de2125baff34c813e85a93cd1d2fcf6c26d66b5e7415f10b": {
                "Name": "shared_alpine_zt_1",
                "EndpointID": "98395ff373decc3ab38879941f637510a6e0afa1892bd3b0337cc996892fa53f",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": "fcf0:a9af:177d:ae37:799d::2/80"
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "zerotier",
            "com.docker.compose.project": "shared"
        }
    }
]

This is the working network:

$ docker network inspect shared_zerotier
[
    {
        "Name": "shared_zerotier",
        "Id": "25d983aaf7eac818e4dcb4f1f4aaf22bac47e69405a3ad7a6709bf5aa9788bd9",
        "Created": "2017-10-02T23:00:28.269292035Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.23.0.0/16",
                    "Gateway": "172.23.0.1"
                },
                {
                    "Subnet": "fcf0:a9af:17ea:c412:57de::/80",
                    "Gateway": "fcf0:a9af:17ea:c412:57de::1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "f74f7086e0eccef468ea8208d15d18ad0ccbfc63c7b93fb625f29325c9fb5b03": {
                "Name": "shared_alpine_zt_1",
                "EndpointID": "799f53ddde7c855b8481659e5a449a0a727cec4b0a010abdf00e204195d80e14",
                "MacAddress": "02:42:ac:17:00:02",
                "IPv4Address": "172.23.0.2/16",
                "IPv6Address": "fcf0:a9af:17ea:c412:57de::2/80"
            },
            "fc36634af3e72697086a3105310c9a4cff205de318c70ff8a98e8baef6a5688c": {
                "Name": "ethereum_parity_1",
                "EndpointID": "bbdb60bf2ec7a77c5b6dc70e674cc209b0479b7434b5495ac51afaa3a082c939",
                "MacAddress": "02:42:ac:17:00:05",
                "IPv4Address": "172.23.0.5/16",
                "IPv6Address": "fcf0:a9af:17ea:c412:57de:0:b37e:f2a9/80"
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "zerotier",
            "com.docker.compose.project": "shared"
        }
    }
]

@janjaapbos
Copy link
Author

@wysenynja my alpine container config at both linux and mac host is similar, and they work fine (can ping each other)

/ # ip -f inet6 route
fc7b:59ab:48c0:b0f4:74a5::/80 dev eth0 metric 256
fe80::/64 dev eth0 metric 256
default via fc7b:59ab:48c0:b0f4:74a5::1 dev eth0 metric 1024
unreachable default dev lo metric -1 error -101
ff00::/8 dev eth0 metric 256
unreachable default dev lo metric -1 error -101
/ #
/ #
/ # ip -f inet6 address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 state UP
inet6 fc7b:59ab:48c0:b0f4:74a5::2/80 scope global flags 02
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe12:2/64 scope link
valid_lft forever preferred_lft forever

@janjaapbos
Copy link
Author

Perhaps you have to check the docker versions? Is there a difference in the version?

@BlinkyStitt
Copy link

laptop: Version 17.09.0-ce-mac33
server: Docker version 17.09.0-ce, build afdb6d4
aws: Docker version 17.06.0-ce, build 02c1d87 (upgrading now)

I still don't understand why I can now ping the ::2 on the server, but not any of the other addresses

@BlinkyStitt
Copy link

Actually, the gateway entry is missing from the config on my server so I'm not sure thats actually a problem. I can only ping the ::2 address. Very strange.

[bwstitt:~/mgmt/tank/shared] [tank] master(+190/-284)+* 4m52s ± docker network inspect shared_zerotier
[
    {
        "Name": "shared_zerotier",
        "Id": "594714072f45aa3c1b020a65b22b77b25085b7cc807939c982ebd0d0d131b5b3",
        "Created": "2017-10-02T16:05:57.579034784-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                },
                {
                    "Subnet": "fcf0:a9af:17a3:c742:eb37::/80"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "19025487bbbfd7baa564b5f6701a837bcf7a89a26064bade306381ee861b89ea": {
                "Name": "monero_monerod_1",
                "EndpointID": "5b4312a3c2259d25847189ac9f39ea0bf624fea16dd4de1c87be29435624c296",
                "MacAddress": "02:42:ac:12:00:07",
                "IPv4Address": "172.18.0.7/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:6115:e08c/80"
            },
            "5b9c8ced353264a3b3e56d3d0a4dbcf47329ca036eab293d0f4b3dc0e73df329": {
                "Name": "ipfs_ipfs_1",
                "EndpointID": "729a90dd72b548a010100cd8497beadf58f3cbb0f636c302139f6893fac16a18",
                "MacAddress": "02:42:ac:12:00:06",
                "IPv4Address": "172.18.0.6/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:6e12:bd36/80"
            },
            "7345180fe54dd5235cac19e0be1fd1b38978b8943147593b53ee6742eb168e27": {
                "Name": "sia_sia_1",
                "EndpointID": "9816e46b8038ccb3dba35993f1b619a0fa82210bb0fb2d24005adc06fa7434b2",
                "MacAddress": "02:42:ac:12:00:08",
                "IPv4Address": "172.18.0.8/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:adc6:2671/80"
            },
            "7d6165962ec50a51c5cf6e7010f781e46a9dcd973cca13a6c7b561070e9c6d25": {
                "Name": "zcash_zcash_1",
                "EndpointID": "2c08c0afdb7666983d59b00ad096cfdf48ab8ab45f09281e0dbcb8919b8a95b0",
                "MacAddress": "02:42:ac:12:00:0a",
                "IPv4Address": "172.18.0.10/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:d7a7:95f0/80"
            },
            "a92d134fd08fc2f3b8aa9c4c76a74b1f3bfa20f571f39bfd46a3a14d742c06c2": {
                "Name": "syncthing_syncthing_1",
                "EndpointID": "c861e679b98101db84b49b46791a3a1b9f642cac52ba95ae06e8773178d8b80a",
                "MacAddress": "02:42:ac:12:00:09",
                "IPv4Address": "172.18.0.9/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:e91f:68f2/80"
            },
            "ac3b4c7679358afcb83271ef4a2012780abc79a14aa53194517b67d67c556235": {
                "Name": "shared_alpine_zt_1",
                "EndpointID": "8bf572c27c32b7e39e8a12d1a8602842a406cdb7464649ad8aa5bd35c9edc9ab",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37::2/80"
            },
            "bf084b0e1815435196c32d0a93a794dcd16280b86819b85b5651ea5a5b412db7": {
                "Name": "frontend_haproxy_zt_1",
                "EndpointID": "cc00d596111d4319fd00829d57e4b8e328a328c55e41170f49d21039e0cf8561",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:c4d:421f/80"
            },
            "cc9a946db4f273e45b665c18b92dbda723f5f0724290f36f39d024d9ee777b16": {
                "Name": "gitolite_gitolite_1",
                "EndpointID": "280d7b6f7d81deba5675acca564ae33ec72c8efae3b7fe1daf92fe16df40e72f",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:d742:14a6/80"
            },
            "ea95d4a749da56db4ff6e91ea1771c659da047473941a7da58def9ca2870d5b7": {
                "Name": "ethereum_parity_1",
                "EndpointID": "80d4e6e1770fea1fa082edcbeb747e1649c76509d18982523f5f4a1881856e83",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": "fcf0:a9af:17a3:c742:eb37:0:b37e:f2a9/80"
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "zerotier",
            "com.docker.compose.project": "shared"
        }
    }
]

@BlinkyStitt
Copy link

traceroute is showing routing as broken.

From my laptop where everything is working as expected:

$ ping6 -c4 fcf0:a9af:17ea:c412:57de::2
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17ea:c412:57de::2
16 bytes from fcf0:a9af:17ea:c412:57de::2, icmp_seq=0 hlim=63 time=3.650 ms
16 bytes from fcf0:a9af:17ea:c412:57de::2, icmp_seq=1 hlim=63 time=3.533 ms
16 bytes from fcf0:a9af:17ea:c412:57de::2, icmp_seq=2 hlim=63 time=9.282 ms
16 bytes from fcf0:a9af:17ea:c412:57de::2, icmp_seq=3 hlim=63 time=6.483 ms

--- fcf0:a9af:17ea:c412:57de::2 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.533/5.737/9.282/2.363 ms


$ traceroute6 fcf0:a9af:17ea:c412:57de::2
traceroute6 to fcf0:a9af:17ea:c412:57de::2 (fcf0:a9af:17ea:c412:57de::2) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17ea:c412:57de::1  36.185 ms  10.257 ms  56.055 ms
 2  fcf0:a9af:17ea:c412:57de::2  3.280 ms  22.197 ms  79.006 ms

From a docker container on my server that I thought was working for one of the ::2, but now I'm not sure whats going on:

/ # ping6 -c4 fcf0:a9af:17ea:c412:57de::2
PING fcf0:a9af:17ea:c412:57de::2 (fcf0:a9af:17ea:c412:57de::2): 56 data bytes
64 bytes from fcf0:a9af:17ea:c412:57de::2: seq=0 ttl=62 time=145.512 ms
64 bytes from fcf0:a9af:17ea:c412:57de::2: seq=1 ttl=62 time=74.499 ms
64 bytes from fcf0:a9af:17ea:c412:57de::2: seq=2 ttl=62 time=78.979 ms
64 bytes from fcf0:a9af:17ea:c412:57de::2: seq=3 ttl=62 time=80.845 ms

--- fcf0:a9af:17ea:c412:57de::2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 74.499/94.958/145.512 ms

/ # traceroute6 fcf0:a9af:17ea:c412:57de::2
traceroute to fcf0:a9af:17ea:c412:57de::2 (fcf0:a9af:17ea:c412:57de::2), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  0.006 ms  0.008 ms  0.003 ms
 2  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  0.003 ms !S  *  0.006 ms !S

From traceroute docs:

Other possible annotations after the time are !H, !N, or
!P (got a host, network or protocol unreachable, respec-
tively), !S or !F (source route failed or fragmentation
needed - neither of these should ever occur and the asso-
ciated gateway is busted if you see one), !X (communica-
tion administratively prohibited), or ! (ICMP unreach-
able code N).

@BlinkyStitt
Copy link

This is definitely related. From my laptop (fcf0:a9af:17b6:4702:db5d::1) to my server (fcf0:a9af:17a3:c742:eb37::/80) where I gave a couple containers IPs but only the ::2 partly works:

$ ping6 fcf0:a9af:17a3:c742:eb37::c4d:421f
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17a3:c742:eb37::c4d:421f
^C
--- fcf0:a9af:17a3:c742:eb37::c4d:421f ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

$ traceroute6 fcf0:a9af:17a3:c742:eb37::c4d:4traceroute6 to fcf0:a9af:17a3:c742:eb37::c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1  10.254 ms  17.548 ms  3.841 ms
 2  fcf0:a9af:17a3:c742:eb37::1  3.566 ms !P  6.112 ms !P  10.194 ms !P


$ ping6 fcf0:a9af:17a3:c742:eb37::2
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17a3:c742:eb37::2
16 bytes from fcf0:a9af:17a3:c742:eb37::2, icmp_seq=0 hlim=63 time=2.723 ms
16 bytes from fcf0:a9af:17a3:c742:eb37::2, icmp_seq=1 hlim=63 time=7.976 ms
16 bytes from fcf0:a9af:17a3:c742:eb37::2, icmp_seq=2 hlim=63 time=6.657 ms
16 bytes from fcf0:a9af:17a3:c742:eb37::2, icmp_seq=3 hlim=63 time=2.299 ms
^C
--- fcf0:a9af:17a3:c742:eb37::2 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 2.299/4.914/7.976/2.452 ms


$ traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute6 to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1  5.682 ms  2.178 ms  2.852 ms
 2  fcf0:a9af:17a3:c742:eb37::1  4.357 ms !P  33.773 ms !P  1.634 ms !P

From the docs:

After the trip time, some additional annotation can be printed: !H, !N, or !P (host, network or protocol unreachable), !S (source route failed), !F (fragmentation needed), !X (communication administratively prohibited), !V (host precedence violation), !C (precedence cutoff in effect), or ! (ICMP unreachable code ). If almost all the probes result in some kind of unreachable, traceroute will give up and exit.

@BlinkyStitt
Copy link

From my laptop to a docker container inside docker-for-mac works:

$ ping6 fcf0:a9af:17ea:c412:57de::2
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17ea:c412:57de::2
16 bytes from fcf0:a9af:17ea:c412:57de::2, icmp_seq=0 hlim=63 time=4.206 ms
...
^C
--- fcf0:a9af:17ea:c412:57de::2 ping6 statistics ---
11 packets transmitted, 11 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.112/8.461/31.982/9.847 ms


$ traceroute6 fcf0:a9af:17ea:c412:57de::2
traceroute6 to fcf0:a9af:17ea:c412:57de::2 (fcf0:a9af:17ea:c412:57de::2) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17ea:c412:57de::1  5.320 ms  7.199 ms  16.895 ms
 2  fcf0:a9af:17ea:c412:57de::2  2.820 ms  15.395 ms  21.636 ms

@BlinkyStitt
Copy link

Still stumped.

fcf0:a9af:17ea:c412:57de::1 is my docker-for-mac VM
fcf0:a9af:17a3:c742:eb37::/80 is my server

On my laptop inside the docker-for-mac VM:

/ # traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  105.661 ms  106.759 ms  101.162 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *
 6  *  *  *
 7  *  *  *
 8  *  *  *
 9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *

On my server:

/ # tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:46:45.082911 IP 10.242.176.103.59719 > 10.242.255.255.21027: UDP, length 69
19:46:45.515682 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33449: UDP, length 24
19:46:50.525709 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33450: UDP, length 24
19:46:55.522750 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33451: UDP, length 24
19:47:00.536413 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33452: UDP, length 24
19:47:05.550436 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33453: UDP, length 24
19:47:05.587103 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:05.587261 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:06.633312 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:06.633487 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:07.673786 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:07.673953 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:10.504823 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33454: UDP, length 24
19:47:10.666627 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor solicitation, who has fe80::4ceb:c2ff:fe71:e70, length 32
19:47:10.747686 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fe80::4ceb:c2ff:fe71:e70, length 24
19:47:15.062791 IP 10.242.176.103.59719 > 10.242.255.255.21027: UDP, length 69
19:47:15.507258 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33455: UDP, length 24
19:47:15.795591 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor solicitation, who has fe80::4ca2:c1ff:fe21:b299, length 32
19:47:15.795625 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor advertisement, tgt is fe80::4ca2:c1ff:fe21:b299, length 24
19:47:20.513083 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33456: UDP, length 24
19:47:25.523430 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33457: UDP, length 24
19:47:30.530364 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33458: UDP, length 24
19:47:35.575428 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33459: UDP, length 24
19:47:35.597537 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:35.597678 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:36.627812 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:36.627946 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:37.673073 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::c4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
19:47:37.673215 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::c4d:421f, source address fe80::4ceb:c2ff:fe71:e70, length 80
19:47:40.547336 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33460: UDP, length 24
19:47:45.048487 IP 10.242.176.103.59719 > 10.242.255.255.21027: UDP, length 69
19:47:45.581256 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33461: UDP, length 24
19:47:50.589429 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33462: UDP, length 24
19:47:55.585646 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33463: UDP, length 24
^[[1;2D19:48:00.592720 IP6 fcf0:a9af:17ea:c412:57de::1.58389 > fcf0:a9af:17a3:c742:eb37::c4d:421f.33464: UDP, length 24

@janjaapbos
Copy link
Author

On the server try a ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f

If that works, do a tcpdump on on eth0 in the fcf0:a9af:17a3:c742:eb37::0c4d:421f container and do on your mac a ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f, and check what you see with tcpdump.

@BlinkyStitt
Copy link

On my server to a container on my server:

$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING fcf0:a9af:17a3:c742:eb37::0c4d:421f(fcf0:a9af:17a3:c742:eb37:0:c4d:421f) 56 data bytes
From fcf0:a9af:17a3:c742:eb37::1 icmp_seq=1 Destination unreachable: Address unreachable

--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping statistics ---
4 packets transmitted, 0 received, +1 errors, 100% packet loss, time 3082ms


$ traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37:0:c4d:421f), 30 hops max, 80 byte packets
 1  tank.stitthappens.com (fcf0:a9af:17a3:c742:eb37::1)  3054.881 ms !H  3054.808 ms !H  3054.796 ms !H

On my laptop:

$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING6(56=40+8+8 bytes) fcf0:a9af:17b6:4702:db5d::1 --> fcf0:a9af:17a3:c742:eb37::c4d:421f

--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

$ traceroute6 fcf0:a9af:17a3:c742:eb37::c4d:421f
traceroute6 to fcf0:a9af:17a3:c742:eb37::c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1  4.301 ms  10.904 ms  2.631 ms
 2  fcf0:a9af:17a3:c742:eb37::1  3062.897 ms !A  3081.672 ms !A  3126.021 ms !A

Here is the tcpdump while that ping/traceroute was happening:

/ # tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:04:50.764308 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:04:50.764386 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:20.764321 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:05:20.764456 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:35.135793 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:36.138648 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:37.162657 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:40.216394 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:41.258636 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:42.282634 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:05:50.764303 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:05:50.764736 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69
21:05:50.794645 IP6 fe80::42:acff:fe12:6 > ip6-allrouters: ICMP6, router solicitation, length 16
21:06:09.346567 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:10.378639 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:11.402634 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:12.429647 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:13.450630 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:14.474653 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:15.503637 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:16.522636 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:17.546658 IP6 fe80::1 > ff02::1:ff4d:421f: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::c4d:421f, length 32
21:06:20.764303 IP6 fe80::42:acff:fe12:9.44904 > ff12::8384.21027: UDP, length 69
21:06:20.764465 IP syncthing_syncthing_1.shared_zerotier.33139 > 172.18.255.255.21027: UDP, length 69

What is really odd is ping/traceroute works from my laptop to my server for a different container:

$ ping6 -c4 fcf0:a9af:17a3:c742:eb37::2
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=2 ttl=64 time=0.150 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=3 ttl=64 time=0.159 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=4 ttl=64 time=0.150 ms

--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.063/0.130/0.159/0.040 ms

$ traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute6 to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2) from fcf0:a9af:17b6:4702:db5d::1, 64 hops max, 12 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1  3.607 ms  2.281 ms  1.627 ms
 2  fcf0:a9af:17a3:c742:eb37::2  1.381 ms  1.096 ms  1.345 ms

@janjaapbos
Copy link
Author

Ok, so if on the server it already does not work to ping its container, you can focus on getting that to work before testing across hosts. So the tcpdump is done on the container where the ping is directed? You see NDP request but no responses.

Can you ping between containers on the same host?

What is the host OS / distro?

Is there a host firewall active?

@BlinkyStitt
Copy link

BlinkyStitt commented Oct 4, 2017 •
edited

Yes, the tcpdump was done inside my haproxy container from this command: docker run --rm -it --net container:frontend_haproxy_zt_1 nicolaka/netshoot

I am able to ping between some containers on the same host (::b37e:f2a9 -> ::2):

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 -c2 fcf0:a9af:17a3:c742:eb37::2
PING fcf0:a9af:17a3:c742:eb37::2(fcf0:a9af:17a3:c742:eb37::2) 56 data bytes
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from fcf0:a9af:17a3:c742:eb37::2: icmp_seq=2 ttl=64 time=0.085 ms

--- fcf0:a9af:17a3:c742:eb37::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.085/0.138/0.192/0.054 ms


# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
 1  shared_alpine_zt_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::2)  0.011 ms  0.005 ms  0.002 ms

It fails for this other host though (::b37e:f2a9 -> ::c4d:421f):

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
traceroute to fcf0:a9af:17a3:c742:eb37::0c4d:421f (fcf0:a9af:17a3:c742:eb37::c4d:421f), 30 hops max, 72 byte packets
 1  ethereum_parity_1.shared_zerotier (fcf0:a9af:17a3:c742:eb37::b37e:f2a9)  3075.682 ms !H  3071.421 ms !H  3071.836 ms !H

# docker run --rm -it --net container:ethereum_parity_1 nicolaka/netshoot ping6 fcf0:a9af:17a3:c742:eb37::0c4d:421f
PING fcf0:a9af:17a3:c742:eb37::0c4d:421f(fcf0:a9af:17a3:c742:eb37::c4d:421f) 56 data bytes
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=1 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=5 Destination unreachable: Address unreachable
From fcf0:a9af:17a3:c742:eb37::b37e:f2a9 icmp_seq=6 Destination unreachable: Address unreachable
^C
--- fcf0:a9af:17a3:c742:eb37::0c4d:421f ping statistics ---
8 packets transmitted, 0 received, +3 errors, 100% packet loss, time 7148ms

Host OS is Fedora 26 with the latest ce version of docker (installed via docker-machine). I've disabled the firewall to simplify the testing.

@janjaapbos
Copy link
Author

Regarding NDP on the host, perhaps this helps:
sysctl -w net.ipv6.conf.all.proxy_ndp=1

@BlinkyStitt
Copy link

Looks promising!

[bwstitt@tank:~] $ sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0

[admin@aws:~] $ sudo sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 0

But changing it to 1 doesn't seem to have made any difference. Pings still fail with the same errors

@BlinkyStitt
Copy link

Latest tcpdump output

[bwstitt:~] $ docker run -it --net host nicolaka/netshoot sysctl net.ipv6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1

[bwstitt:~] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::4
traceroute to fcf0:a9af:17a3:c742:eb37::4 (fcf0:a9af:17a3:c742:eb37::4), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  299.604 ms  312.555 ms  338.743 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *^C

[root@tank] # docker run -it --net host nicolaka/netshoot sysctl nev6.conf.all.proxy_ndp
net.ipv6.conf.all.proxy_ndp = 1

[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:05:36.083679 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33435: UDP, length 24
02:05:36.083873 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:36.327194 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33436: UDP, length 24
02:05:36.327332 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::4, length 80
02:05:39.686508 IP 10.242.176.103.54421 > 10.242.255.255.21027: UDP, length 69
02:05:41.450627 IP6 fe80::4ca2:c1ff:fe21:b299 > fcf0:a9af:17ea:c412:57de::1: ICMP6, neighbor solicitation, who has fcf0:a9af:17ea:c412:57de::1, length 32
02:05:41.562750 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33438: UDP, length 24
02:05:41.756616 IP6 fcf0:a9af:17ea:c412:57de::1 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fcf0:a9af:17ea:c412:57de::1, length 24
02:05:46.577343 IP6 fcf0:a9af:17ea:c412:57de::1.47406 > fcf0:a9af:17a3:c742:eb37::4.33439: UDP, length 24
02:05:46.634252 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:46.634393 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:46.790118 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor solicitation, who has fe80::4ca2:c1ff:fe21:b299, length 32
02:05:46.790142 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor advertisement, tgt is fe80::4ca2:c1ff:fe21:b299, length 24
02:05:47.710453 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:47.710635 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:48.733475 IP6 fe80::4ceb:c2ff:fe71:e70 > fcf0:a9af:17a3:c742:eb37::4: ICMP6, neighbor solicitation, who has fcf0:a9af:17a3:c742:eb37::4, length 32
02:05:48.733608 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, destination unreachable, beyond scope fcf0:a9af:17a3:c742:eb37::4, source address fe80::4ceb:c2ff:fe71:e70, length 80
02:05:51.690628 IP6 fe80::4ca2:c1ff:fe21:b299 > fe80::4ceb:c2ff:fe71:e70: ICMP6, neighbor solicitation, who has fe80::4ceb:c2ff:fe71:e70, length 32
02:05:51.909489 IP6 fe80::4ceb:c2ff:fe71:e70 > fe80::4ca2:c1ff:fe21:b299: ICMP6, neighbor advertisement, tgt is fe80::4ceb:c2ff:fe71:e70, length 24

And here is a successful traceroute for a different container on the same host:

[bwstitt@laptop] $ docker run -it --net host nicolaka/netshoot traceroute6 fcf0:a9af:17a3:c742:eb37::2
traceroute to fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2), 30 hops max, 72 byte packets
 1  fcf0:a9af:17a3:c742:eb37::1 (fcf0:a9af:17a3:c742:eb37::1)  245.199 ms  *  324.855 ms
 2  fcf0:a9af:17a3:c742:eb37::2 (fcf0:a9af:17a3:c742:eb37::2)  305.673 ms  309.446 ms  309.176 ms

[root@tank] # docker run -it --net host nicolaka/netshoot tcpdump -i zt0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:13:54.875535 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33435: UDP, length 24
02:13:54.875678 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.145734 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33437: UDP, length 24
02:14:00.145908 IP6 fcf0:a9af:17a3:c742:eb37::1 > fcf0:a9af:17ea:c412:57de::1: ICMP6, time exceeded in-transit for fcf0:a9af:17a3:c742:eb37::2, length 80
02:14:00.459565 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33438: UDP, length 24
02:14:00.459606 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33438, length 80
02:14:00.763468 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33439: UDP, length 24
02:14:00.763519 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33439, length 80
02:14:01.081223 IP6 fcf0:a9af:17ea:c412:57de::1.49226 > fcf0:a9af:17a3:c742:eb37::2.33440: UDP, length 24
02:14:01.081259 IP6 fcf0:a9af:17a3:c742:eb37::2 > fcf0:a9af:17ea:c412:57de::1: ICMP6, destination unreachable, unreachable port, fcf0:a9af:17a3:c742:eb37::2 udp port 33440, length 80

@jgentes
Copy link

jgentes commented Jun 14, 2019

Looks like zerotier/zerotier-containerized is gone :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment