This is an outdated draft that was used to review the content with FreeBSD contributors. For the latest revision, vist http://shawndebnath.com/articles/2016/03/27/freebsd-jails-with-vlan-howto.html.
This article discusses how to set up jails on a FreeBSD 11-CURRENT system utilizing VIMAGE (aka VNET) to provide a virtualized independent network stack for each jail with support for VLAN tagging.
- You have a machine installed with FreeBSD 11-CURRENT on ZFS.
- We will be building world and kernel and using that as the base for the jails. Hence basic knowledge of FreeBSD system administration is assumed. If you've never compiled and installed a FreeBSD base system and kernel, this article may be hard to follow. Refer to the FreeBSD Handbook, especially chapter 8: 'Configuring the FreeBSD Kernel' and chapter 23: 'Updating and Upgrading FreeBSD'.
- Your host's ethernet interface is em0.
- Your IP network is 192.168.6.0/24 with gateway at 192.168.6.1.
- The host will be assigned IP 192.168.6.66.
- The guest jails will be assigned IPs in the range 192.168.6.100-254.
- VLAN ID for all network interfaces will be 6.
- Jails will be stored in ZFS datasets under /jail directory.
On the host system, we need to re-compile the kernel to include EPAIR(4) and IF_BRIDGE(4) devices and the VIMAGE option to enable virtualized network stack capabilities. We also enable NULLFS so that we can mount ports and the src tree inside jails.
# Virtual networking for jail
options VIMAGE
device epair
device if_bridge
# Enable nullFS to mount src and port directories
options NULLFS
It's not a bad idea to fetch the latest 11-CURRENT sources to get the latest fixes and rebuild and install world along with kernel at this point.
First, let's set up the networking configuration. To enable VLANs, we need to create a cloned interface on which will set the VLAN parameters. We will also create a bridge that the paired networked interfaces for the jails will be trunked on.
ifconfig_em0="up"
cloned_interfaces="vlan0 bridge0"
ifconfig_vlan0="inet 192.168.6.66 netmask 255.255.255.0 vlan 6 vlandev em0"
defaultrouter="192.168.6.1"
ifconfig_bridge0="addm em0 up"
This setup will enable the host to access the network via vlan0 interface through the em0 physical adapater. As each paired interface will allow the guest to access the network by having its traffic flow to em0 via bridge0 skipping vlan0 completely.
Next, we need to enable jails support:
jail_enable="YES"
jail_list=""
We also need to restrict some of the services on the host so that they don't interfere with the jails:
dumpdev="AUTO" # Set to AUTO to enable crash dump, otherwise NO
rpcbind_enable="NO" # Disable the RPC daemon
cron_flags="$cron_flags -J 15" # Prevent jails running cron jobs at same time
syslogd_flags="-ss" # Disable syslogd listening for incoming conns
sendmail_enable="NONE" # Completely disable sendmail
clear_tmp_enable="YES" # Clear /tmp at startup
inetd_flags="-wW -a $EXT_IP" # Restrict inetd to not interfere with jails
FreeBSD implements jails v2 which requires configuration to be in /etc/jail.conf. To get started, we will create /etc/jail.conf with the default configuration applicable to all jails:
# Jail configuration - /etc/jail.conf
allow.raw_sockets = "1";
allow.set_hostname = "0";
allow.sysvipc = "1";
allow.mount.devfs;
host.hostname = "${name}";
path = "/jail/${name}";
mount.fstab = "/etc/jails/fstab.${name}";
mount.devfs = "1";
devfs_ruleset = "4";
exec.consolelog = "/var/log/jail_${name}_console.log";
Jail specific configuration will be added to /etc/jail.conf as we create each jail below.
Each jail will have it's own fstab which will be placed under /etc/jails/, so create that directory:
sudo mkdir /etc/jails
Reboot your system or restart networking manually:
sudo service netif restart
sudo service routing restart
Running ifconfig should result in output similar to this (lo0 omitted for brevity):
em0:flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
ether 68:05:ca:36:3c:26
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 68:05:ca:36:3c:26
inet 192.168.6.66 netmask 0xffffff00 broadcast 192.168.6.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 6 parent interface: em0
groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:0b:47:0d:33:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
For demonstration purposes, we will create a jail called webserver whose intent is to run a web server. The installation and configuration of the web server is out of scope for this guide.
We will create a ZFS dataset with quota for 2GB (substitute your zpool name accordingly):
sudo zfs create zroot/webserver
sudo zfs set mountpoint=/jail/webserver zroot/webserver
sudo zfs set quota=2g zroot/webserver
We will use the pre-built system binaries to populate our jail as follows:
cd /usr/src
sudo make installworld DESTDIR=/jail/webserver
sudo make distribution DESTDIR=/jail/webserver
If you chose to use a release distribution for the jail instead of 11-CURRENT binaries, extract the content from an appropriate ISO image accordingly. You may also wish to build the binaries from sources other than 11-CURRENT.
We set up the jail's rc.conf similar to the host:
hostname="webserver"
sendmail_enable="NONE" # Completely disable sendmail
inetd_flags="-wW -a 192.168.6.100" # Restrict inetd to not interfere with jails
rpcbind_enable="NO" # Disable the RPC daemon
cron_flags="$cron_flags -J 15" # Prevent jails running cron jobs at same time
syslogd_flags="-ss" # Disable syslogd listening for incoming conns
Create a new file /etc/jails/fstab.webserver and set it's contents to:
/usr/src /jail/webserver/usr/src nullfs rw 0 0
/usr/ports /jail/webserver/usr/ports nullfs rw 0 0
Finally, we need to set up the jail specific configuration in jail.conf. Here, we are assigning 192.168.6.100 to the jail and tagging it for VLAN ID 6. We specify that the jail will be using its own virtual networking stack with the vnet command and set the vnet interface is set to epair100b.
# VIMAGE jail with upnp-based dlna server with its own NIC address
webserver {
$if = "100";
$ip_addr = "192.168.6.${if}"; # Jail ip address
$ip_route = "192.168.6.1"; # Gateway or host's ip address
$mask = "255.255.255.0"; # Netmask
vnet;
vnet.interface = "epair${if}b";
# Commands to run on host before jail is created
exec.prestart = "ifconfig epair${if} create up";
exec.prestart += "ifconfig epair${if}a up";
exec.prestart += "ifconfig bridge0 addm epair${if}a up";
# Commands to run in jail after it is created
exec.start = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${if}b up";
exec.start += "/sbin/ifconfig vlan${if} create";
exec.start += "/sbin/ifconfig vlan${if} ${ip_addr} netmask ${mask} vlan 6 vlandev epair${if}b up";
exec.start += "/sbin/route add default ${ip_route}";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 deletem epair${if}a";
exec.poststop += "ifconfig epair${if}a destroy";
persist;
}
Because the jail has it's own networking stack, the procedure to set the VLAN interface is similar to how we set it up for the host with a couple of caveats:
- Prior to the jail being created (see exec.start), we create the epair(4) interfaces and add the epair100a interface to bridge0 on the host. This is the virtualized network adapter for the jail.
- One the jail is up (see exec.start), we activate the lo0 and epair100b interfaces and proceed on to creating the cloned vlan interface on the epair100b interface similar to how we did for the host.
sudo service jail start webserver
If all went well, you should see the jail listed (not IP Address is not shown for VNET based jails):
$ jls
JID IP Address Hostname Path
1 webserver /jail/webserver
On the host, ifconfig should report the newly created epair100a interface along with it being a member of bridge0:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:0b:47:0d:33:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair100a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
epair100a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:70:00:07:0a
inet6 fe80::ff:70ff:fe00:70a%epair100a prefixlen 64 scopeid 0x7
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
And ifconfig output inside the jail should report the corresponding epair100b interface and the configured vlan0 interface:
epair100b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:c0:00:08:0b
inet6 fe80::ff:c0ff:fe00:80b%epair100b prefixlen 64 tentative scopeid 0x2
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:ff:c0:00:08:0b
inet6 fe80::ff:c0ff:fe00:80b%vlan100 prefixlen 64 tentative scopeid 0x3
inet 192.168.6.100 netmask 0xffffff00 broadcast 192.168.6.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
vlan: 6 parent interface: epair100b
groups: vlan
Great write up! Any idea how this would reflect to Iocage jails? There is a config.json per jail which looks similar but... I just cant get it to work.
Cheers