triton_overlay_issue.md

Here is the fabric NIC on a Debian 10 bhyve VM on cn01:

root@k8s-worker-2:~# netstat -nr | head -2; netstat -nr | grep 192.168.1
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.128.0   0.0.0.0         255.255.252.0   U         0 0          0 net1

Here's the full routing table:

root@k8s-worker-2:~# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 net0
10.42.0.0       10.42.0.0       255.255.255.0   UG        0 0          0 flannel.1
10.42.1.108     0.0.0.0         255.255.255.255 UH        0 0          0 cali4c9f66466a5
10.42.1.109     0.0.0.0         255.255.255.255 UH        0 0          0 cali5f29d27729d
10.42.1.110     0.0.0.0         255.255.255.255 UH        0 0          0 calib324dfbd949
10.42.1.111     0.0.0.0         255.255.255.255 UH        0 0          0 cali370448994b3
10.42.1.112     0.0.0.0         255.255.255.255 UH        0 0          0 cali2f886aeb28a
10.42.1.113     0.0.0.0         255.255.255.255 UH        0 0          0 caliadf71d5c0b1
10.42.1.114     0.0.0.0         255.255.255.255 UH        0 0          0 calibbc4091a8bb
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
192.168.0.0     0.0.0.0         255.255.254.0   U         0 0          0 net0
192.168.128.0   0.0.0.0         255.255.252.0   U         0 0          0 net1

And here is a fabric NIC on a Triton volume on the HN:

[root@bb6922dd-22c7-c83a-93e0-bb311ae68947 /zones/bb6922dd-22c7-c83a-93e0-bb311ae68947/data]# netstat -nr | head -3; netstat -nr | grep 192.168.1

Routing Table: IPv4
  Destination            Gateway          Flags  Ref     Use     Interface 
default              192.168.128.1        UG        2        598 net0      
192.168.128.0        192.168.130.149      U         3          0 net0

The bhyve VM on cn01 can't ping the Triton volume over the fabric network:

root@k8s-worker-2:~# ping 192.168.130.149
PING 192.168.130.149 (192.168.130.149) 56(84) bytes of data.
From 192.168.130.142 icmp_seq=1 Destination Host Unreachable
From 192.168.130.142 icmp_seq=2 Destination Host Unreachable
From 192.168.130.142 icmp_seq=3 Destination Host Unreachable
From 192.168.130.142 icmp_seq=4 Destination Host Unreachable
From 192.168.130.142 icmp_seq=5 Destination Host Unreachable
From 192.168.130.142 icmp_seq=6 Destination Host Unreachable
^C

Here's the bhyve VM's NICS returned from mdata-get sdc:nics:

[
    {
        "interface": "net0",
        "mac": "90:b8:d0:49:fe:15",
        "vlan_id": 0,
        "nic_tag": "external",
        "gateway": "192.168.0.1",
        "gateways": [
            "192.168.0.1"
        ],
        "netmask": "255.255.254.0",
        "ip": "192.168.0.172",
        "ips": [
            "192.168.0.172/23"
        ],
        "network_uuid": "36cc5b70-a8bf-40e4-9df8-a244fa45be26",
        "model": "virtio",
        "mtu": 1500,
        "primary": true
    },
    {
        "interface": "net1",
        "mac": "90:b8:d0:5d:87:27",
        "vlan_id": 2,
        "nic_tag": "sdc_overlay/14876120",
        "gateway": "192.168.128.1",
        "gateways": [
            "192.168.128.1"
        ],
        "netmask": "255.255.252.0",
        "ip": "192.168.130.142",
        "ips": [
            "192.168.130.142/22"
        ],
        "network_uuid": "e0fb4ea4-3119-4b22-902e-40328c3ab418",
        "model": "virtio",
        "mtu": 8500
    }
]

Which correctly has the gateway set on net1 but the cloud-init file does not:

root@k8s-worker-2:~# cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback
    dns-nameservers 192.168.0.166 208.67.220.220 1.0.0.1

auto net0
iface net0 inet static
    address 192.168.0.172/23
    gateway 192.168.0.1
    mtu 1500

auto net1
iface net1 inet static
    address 192.168.130.142/22
    mtu 8500