vnet byve issues

gistfile1.txt

My bhyve-VM cannot converse with a webserver on the host.  Webservers not on the host are OK.

This is a firewall issue.

Host NICS:

[dan@slocum:~] $ ifconfig -u
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
	ether 00:1b:21:39:a9:c5
...
	inet 10.55.0.24 netmask 0xffffffff broadcast 10.55.0.24
...
	media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
	groups: pflog
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 3e:18:c0:b5:b1:c5
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: ix2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 3 priority 128 path cost 2000
	groups: bridge vm-switch viid-4c918@
	nd6 options=1<PERFORMNUD>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: vmnet-myguest-0-public
	options=80000<LINKSTATE>
	ether 58:9c:fc:10:07:29
	groups: tap vm-port
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	Opened by PID 52659

The bhyve-vm is on 10.55.0.123

It cannot reach my package server at 10.55.0.24 (which is actually an Nginx proxy to the real server). The problem is firewall rules.

The package server is in a jail. That is relevant to the firewall rules.

This is in the vm:

dvl@testing:~ % fetch http://google.ca
fetch: http://google.ca: size of remote file is not known
google.ca                                               12 kB   48 MBps    00s

Reaching websites, NOT on this host, works.

Reaching a website on this host, does not, despite pings being OK:

dvl@testing:~ % ping fedex.int.unixathome.org
PING fedex.int.unixathome.org (10.55.0.24): 56 data bytes
64 bytes from 10.55.0.24: icmp_seq=0 ttl=64 time=0.222 ms
^C
--- fedex.int.unixathome.org ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.222/0.222/0.222/0.000 ms


dvl@testing:~ % fetch http://fedex.int.unixathome.org
fetch: http://fedex.int.unixathome.org: Operation timed out


While the above is going on, you can see this traffic being blocked on pflog:

13:46:34.080166 IP 10.55.0.24.80 > 10.55.0.123.50781: Flags [S.], seq 3875793680, ack 2128927570, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2654640284 ecr 2263288606], length 0
13:46:35.140560 IP 10.55.0.24.80 > 10.55.0.123.50781: Flags [S.], seq 156298239, ack 2128927570, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2654641345 ecr 2263289674], length 0


That means the website is getting the requests from the VM and is replying back, but that reply is blocked.

Here are all the skips now in my pf ruleset:

[dan@slocum:~] $ grep skip /etc/pf.conf
set skip on lo0
set skip on epair
set skip on bridge0
set skip on tap1
set skip on vm-public
set skip on bridge
set skip on tap

Firewall rules for the guest VM:

[dan@slocum:~] $ grep 10.55.0.123 /etc/pf.conf
MYGUEST       = "10.55.0.123"
[dan@slocum:~] $ grep MYGUEST /etc/pf.conf
MYGUEST       = "10.55.0.123"
pass out quick from $MYGUEST flags S/SA keep state
pass quick to   $MYGUEST 
table <RESTRICTED_JAILS>      { ... $MYGUEST, ... }
table <MAIL_SOURCES>          { ... $MYGUEST }
[dan@slocum:~] $ 


[dan@slocum:~] $ grep RESTRICTED_JAILS /etc/pf.conf
pass out quick on $LAN_IF inet proto tcp from <RESTRICTED_JAILS> to <BACULA_STORAGE> port bacula-sd  flags S/SA keep state
pass out quick on $LAN_IF inet proto tcp from <RESTRICTED_JAILS> to any              port $WEB_PORTS flags S/SA keep state
pass out quick on $LAN_IF net proto tcp from <RESTRICTED_JAILS>  to any                              flags S/SA keep state
pass out quick            net proto tcp from <RESTRICTED_JAILS>  to $PKG_SERVER      port $WEB_PORTS flags S/SA keep state
pass out quick on $LAN_IF net proto tcp from <RESTRICTED_JAILS>  to $PKG_SERVER      port $WEB_PORTS flags S/SA keep state




NOTES:

10.55.0.24 is on ix0
10.55.0.123 is not shown above; it's in the VM:

dvl@testing:~ % netstat -nr -4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.55.0.1          UGS      vtnet0
10.0.0.0/8         link#1             U        vtnet0
10.55.0.123        link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
dvl@testing:~ % 


dvl@testing:~ % ifconfig 
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
	ether 58:9c:fc:0f:fc:b4
	inet 10.55.0.123 netmask 0xff000000 broadcast 10.255.255.255
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
dvl@testing:~ % 



The traffic from the VM is getting to the webserver, but the replies are being blocked.

This is helpful:

[dan@slocum:~] $ sudo tcpdump -n -e -ttt -i pflog0
 00:00:00.000000 rule 0/0(match): block out on ix2: 10.55.0.24.80 > 10.55.0.123.34318: Flags [S.], seq 1701654419, ack 1386482183, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1797144388 ecr 1964785594], length 0


It is useful because it's a 'out on ix2' situation.  

Let's add in in this rule:

pass out quick on ix2 inet proto tcp from 10.55.0.24 port 80 to 10.55.0.123 flags S/SA keep state


00:00:01.058011 rule 0/0(match): block out on ix2: 10.55.0.24.80 > 10.55.0.123.13882: Flags [S.], seq 2041794959, ack 198628267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3016114281 ecr 3758404640], length 0