Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

This is an outdated draft that was used to review the content with FreeBSD contributors. For the latest revision, vist http://shawndebnath.com/articles/2016/03/27/freebsd-jails-with-vlan-howto.html.


FreeBSD Jails with VLAN HOWTO

This article discusses how to set up jails on a FreeBSD 11-CURRENT system utilizing VIMAGE (aka VNET) to provide a virtualized independent network stack for each jail with support for VLAN tagging.

Prerequisites

  • You have a machine installed with FreeBSD 11-CURRENT on ZFS.
  • We will be building world and kernel and using that as the base for the jails. Hence basic knowledge of FreeBSD system administration is assumed. If you've never compiled and installed a FreeBSD base system and kernel, this article may be hard to follow. Refer to the FreeBSD Handbook, especially chapter 8: 'Configuring the FreeBSD Kernel' and chapter 23: 'Updating and Upgrading FreeBSD'.

Assumptions

  • Your host's ethernet interface is em0.
  • Your IP network is 192.168.6.0/24 with gateway at 192.168.6.1.
  • The host will be assigned IP 192.168.6.66.
  • The guest jails will be assigned IPs in the range 192.168.6.100-254.
  • VLAN ID for all network interfaces will be 6.
  • Jails will be stored in ZFS datasets under /jail directory.

Configure the host system

Reuild world and kernel

On the host system, we need to re-compile the kernel to include EPAIR(4) and IF_BRIDGE(4) devices and the VIMAGE option to enable virtualized network stack capabilities. We also enable NULLFS so that we can mount ports and the src tree inside jails.

# Virtual networking for jail
options         VIMAGE
device          epair
device          if_bridge

# Enable nullFS to mount src and port directories
options         NULLFS

It's not a bad idea to fetch the latest 11-CURRENT sources to get the latest fixes and rebuild and install world along with kernel at this point.

Update rc.conf

First, let's set up the networking configuration. To enable VLANs, we need to create a cloned interface on which will set the VLAN parameters. We will also create a bridge that the paired networked interfaces for the jails will be trunked on.

ifconfig_em0="up"
cloned_interfaces="vlan0 bridge0"
ifconfig_vlan0="inet 192.168.6.66 netmask 255.255.255.0 vlan 6 vlandev em0"
defaultrouter="192.168.6.1"
ifconfig_bridge0="addm em0 up"

This setup will enable the host to access the network via vlan0 interface through the em0 physical adapater. As each paired interface will allow the guest to access the network by having its traffic flow to em0 via bridge0 skipping vlan0 completely.

Next, we need to enable jails support:

jail_enable="YES"
jail_list=""

We also need to restrict some of the services on the host so that they don't interfere with the jails:

dumpdev="AUTO"                  # Set to AUTO to enable crash dump, otherwise NO
rpcbind_enable="NO"             # Disable the RPC daemon
cron_flags="$cron_flags -J 15"  # Prevent jails running cron jobs at same time
syslogd_flags="-ss"             # Disable syslogd listening for incoming conns
sendmail_enable="NONE"          # Completely disable sendmail
clear_tmp_enable="YES"          # Clear /tmp at startup
inetd_flags="-wW -a $EXT_IP"    # Restrict inetd to not interfere with jails

Configure jails support

FreeBSD implements jails v2 which requires configuration to be in /etc/jail.conf. To get started, we will create /etc/jail.conf with the default configuration applicable to all jails:

# Jail configuration - /etc/jail.conf

allow.raw_sockets = "1";
allow.set_hostname = "0";
allow.sysvipc = "1";
allow.mount.devfs;
  
host.hostname  = "${name}";
path  = "/jail/${name}";
mount.fstab  = "/etc/jails/fstab.${name}";
mount.devfs  = "1";
devfs_ruleset  = "4";
exec.consolelog  = "/var/log/jail_${name}_console.log";

Jail specific configuration will be added to /etc/jail.conf as we create each jail below.

Each jail will have it's own fstab which will be placed under /etc/jails/, so create that directory:

sudo mkdir /etc/jails

Reboot your system or restart networking manually:

sudo service netif restart
sudo service routing restart

Running ifconfig should result in output similar to this (lo0 omitted for brevity):

em0:flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
    ether 68:05:ca:36:3c:26
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 68:05:ca:36:3c:26
    inet 192.168.6.66 netmask 0xffffff00 broadcast 192.168.6.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 6 parent interface: em0
    groups: vlan

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:0b:47:0d:33:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge 
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000

Create the jail

For demonstration purposes, we will create a jail called webserver whose intent is to run a web server. The installation and configuration of the web server is out of scope for this guide.

Create ZFS dataset for the jail

We will create a ZFS dataset with quota for 2GB (substitute your zpool name accordingly):

sudo zfs create zroot/webserver
sudo zfs set mountpoint=/jail/webserver zroot/webserver
sudo zfs set quota=2g zroot/webserver

Populate filesystem

We will use the pre-built system binaries to populate our jail as follows:

cd /usr/src
sudo make installworld DESTDIR=/jail/webserver
sudo make distribution DESTDIR=/jail/webserver

If you chose to use a release distribution for the jail instead of 11-CURRENT binaries, extract the content from an appropriate ISO image accordingly. You may also wish to build the binaries from sources other than 11-CURRENT.

Create the jail's rc.conf

We set up the jail's rc.conf similar to the host:

hostname="webserver"
sendmail_enable="NONE"          # Completely disable sendmail
inetd_flags="-wW -a 192.168.6.100"    # Restrict inetd to not interfere with jails
rpcbind_enable="NO"             # Disable the RPC daemon
cron_flags="$cron_flags -J 15"  # Prevent jails running cron jobs at same time
syslogd_flags="-ss"             # Disable syslogd listening for incoming conns

Create the jail's fstab on host

Create a new file /etc/jails/fstab.webserver and set it's contents to:

/usr/src              /jail/webserver/usr/src            nullfs      rw     0     0
/usr/ports            /jail/webserver/usr/ports          nullfs      rw     0     0

Create the jail specific configuration in jail.conf

Finally, we need to set up the jail specific configuration in jail.conf. Here, we are assigning 192.168.6.100 to the jail and tagging it for VLAN ID 6. We specify that the jail will be using its own virtual networking stack with the vnet command and set the vnet interface is set to epair100b.

# VIMAGE jail with upnp-based dlna server with its own NIC address
webserver {
  $if  = "100";
  $ip_addr  = "192.168.6.${if}";   # Jail ip address
  $ip_route  = "192.168.6.1";  # Gateway or host's ip address
  $mask = "255.255.255.0";  # Netmask
  vnet;
  vnet.interface  = "epair${if}b";

  # Commands to run on host before jail is created
  exec.prestart  = "ifconfig epair${if} create up";
  exec.prestart  += "ifconfig epair${if}a up";
  exec.prestart  += "ifconfig bridge0 addm epair${if}a up";

  # Commands to run in jail after it is created
  exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";
  exec.start  += "/sbin/ifconfig epair${if}b up";
  exec.start  += "/sbin/ifconfig vlan${if} create";
  exec.start  += "/sbin/ifconfig vlan${if} ${ip_addr} netmask ${mask} vlan 6 vlandev epair${if}b up";
  exec.start  += "/sbin/route add default ${ip_route}";
  exec.start  += "/bin/sh /etc/rc";

  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.poststop  = "ifconfig bridge0 deletem epair${if}a";
  exec.poststop  += "ifconfig epair${if}a destroy";
  persist;
}

Because the jail has it's own networking stack, the procedure to set the VLAN interface is similar to how we set it up for the host with a couple of caveats:

  • Prior to the jail being created (see exec.start), we create the epair(4) interfaces and add the epair100a interface to bridge0 on the host. This is the virtualized network adapter for the jail.
  • One the jail is up (see exec.start), we activate the lo0 and epair100b interfaces and proceed on to creating the cloned vlan interface on the epair100b interface similar to how we did for the host.

Start the jail

sudo service jail start webserver

If all went well, you should see the jail listed (not IP Address is not shown for VNET based jails):

$ jls
   JID  IP Address      Hostname                      Path
     1                  webserver                     /jail/webserver

On the host, ifconfig should report the newly created epair100a interface along with it being a member of bridge0:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>     metric 0 mtu 1500
	ether 02:0b:47:0d:33:00
	nd6 options=9<PERFORMNUD,IFDISABLED>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair100a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 1 priority 128 path cost 2000000

epair100a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:ff:70:00:07:0a
	inet6 fe80::ff:70ff:fe00:70a%epair100a prefixlen 64 scopeid 0x7 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 

And ifconfig output inside the jail should report the corresponding epair100b interface and the configured vlan0 interface:

epair100b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:ff:c0:00:08:0b
	inet6 fe80::ff:c0ff:fe00:80b%epair100b prefixlen 64 tentative scopeid 0x2 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 

vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:ff:c0:00:08:0b
	inet6 fe80::ff:c0ff:fe00:80b%vlan100 prefixlen 64 tentative scopeid 0x3 
	inet 192.168.6.100 netmask 0xffffff00 broadcast 192.168.6.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	vlan: 6 parent interface: epair100b
	groups: vlan 
@ezraholm50

This comment has been minimized.

Copy link

ezraholm50 commented Feb 10, 2019

Great write up! Any idea how this would reflect to Iocage jails? There is a config.json per jail which looks similar but... I just cant get it to work.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.