A short description on how to use iohyve on FreeNAS 9.10

freenas-iohyve.md

Introduction

FreeNAS 9.10 is based on FreeBSD 10.3 and as such, supports the BSD hypervisor bhyve. There are different frontends for bhyve, one of them is iohyve which is included in FreeNAS 9.10. This document describes how to setup iohyve for operation on FreeNAS 9.10 and basic VM management tasks.

Setting up Iohyve

Iohyve is already installed in FreeNAS 9.10, so it only needs to be enabled. This is done by adding the following lines to /conf/base/etc/rc.conf:

iohyve_enable="YES"
iohyve_flags="kmod=1 net=igb0 pool=storage-volume"

The first line enables iohyve generally, while the second line provides some configuration. Specifically, it specifies that iohyve should load the required kernel modules itself, use igb0 as bridge interface for all VMs and use the zpool storage-volume for vm storage.

This change only becomes active after a reboot. Alternatively, iohyve setup can be run manually.

Creating and installing the first VM

First, a new iohyve VM has to be created:

# iohyve create debianvm 20G

This creates a VM called debianvm with one disk which has a size of 20 Gigabytes. A few more settings for the vm need to be adjusted:

# iohyve set debianvm loader=grub-bhyve
# iohyve set debianvm os=debian
# iohyve set debianvm ram=2G
# iohyve set debianvm cpu=2

This tells iohyve to run the vm using grub-bhyve a tool which simulates grub before the actual virtualization starts. It further tells iohyve that Debian is running in the VM, presumably to allow for some optimization regarding simulated hardware. Then, it tells iohyve that it should allocate 2 Gigabytes of RAM and 2 virtual CPUs to the VM.

Now, download and verify an installer image for the distribution of your choice. Then, add it for usage with iohyve:

# iohyve cpiso debian-8.6.0-amd64-netinst.iso

Then in one terminal, attach to the VM's console:

# iohyve console debianvm

and start the installation from another terminal:

# iohyve install debianvm debian-8.6.0-amd64-netinst.iso

The console should show a boot menu and later the installer interface. Take care to install the grub config onto the first partition of the disk.

Cloning the VM to create a second VM

The VM which was just installed can be cloned quickly to create a new VM without performing the installation again:

# iohyve clone debianvm debian-foo

This creates a new VM debian-foo which is a clone of debianvm. By default, it shares its console and networking interface with debianvm which will not work if both will run at the same time. To fix this, configure a dedicated console and networking interface for the new VM:

# iohyve set debian-bar con=nmdm1
# iohyve set debian-bar tap=tap1

To find a usable console when you have many vms, you can use the following snippet which will show you the ones currently in use:

# WHAT=tap; iohyve list | tail -n +2 | cut -d ' ' -f 1 | while read vm; do iohyve get $vm $WHAT | tail -n +2; done | sort

with WHAT=tap or WHAT=con.

To start the cloned vm use:

# iohyve start debian-bar

To ensure that it starts on every boot set the boot variable:

# iohyve set debian-bar boot=1

Other Lifecycle operations

To shutdown a VM, the stop command can be used:

# iohyve stop debian-bar

If the machine is not responding, it can be forced to stop:

# iohyve destroy debian-bar

Also, additional disks can be provided to the VM, e.g. the following command adds a 100G volume to the VM:

# iohyve add debian-bar 100G

Updating FreeNAS

After a FreeNAS update, the change to /conf/base/etc/rc.conf is very likely gone. If it is, iohyve and the VMs associated with it will not run. To restore it, perform the following steps:

First, append the following lines to /conf/base/etc/rc.conf again, make sure to adjust pool name and netif name if needed:

iohyve_enable="YES"
iohyve_flags="kmod=1 net=igb0 pool=storage-volume"

Then, propagate this change to /etc/rc.conf, after reviewing that there is no other difference between the files:

# diff -au /conf/base/etc/rc.conf /etc/rc.conf
# cp /conf/base/etc/rc.conf /etc/rc.conf

After doing so, you should be able to start iohyve with its initscript:

# /usr/local/etc/rc.d/iohyve start

And all VMs should boot up again.

@cfra Thanks for this! This is particularly useful as FreeNAS 9.10 no longer supports VirtualBox. Readers: take note of man iohyve as well.

To simplify clone example, the tap/con settings are handled by iohyve clone -r <name> <clonename>, which will find the next available values for tap/con.