I have a problem. I need to run some Docker containers for work. Docker, as you may know, has a design that is tightly coupled to Linux. This is rather unfortunate since my desktop is running FreeBSD, and my laptop is running its distant cousin, macOS. Normally I'd just do such things on my Mac, but 1) my desktop has a lot more storage, and 2) the Docker images in question are amd64 already. Also I just wanted a challenge, so let's set up a VM to run Docker on FreeBSD.
Aside in case you're wondering why Docker runs on your Mac: Docker Desktop actually ships a hypervisor, HyperKit, to run Linux. If I'm not mistaken, this is actually based on a macOS port of bhyve. Read more here: https://medium.com/@hintcnuie/the-difference-between-xhyve-and-hyperkit-67a9f378ab97.
bhyve is a hypervisor for FreeBSD (and a few other platforms). There is no slick GUI or anything; it's all CLI-driven. This makes for a steep initial learning curve. Once you have everything set up though, it's great.
This is not an exhaustive guide, and I'm sure I missed a few things; please leave a comment if you notice anything off. The objective is to get a simple Debian base system installed with console access.
NOTE: All commands must be run as root.
bhyve is part of the base system since FreeBSD 10.0. See the Handbook for a quick overview. All you need to do is make sure the kernel module is loaded. We will also install a helper package that makes things a little easier to work with.
kldload vmm
sysrc vmm_enable="YES"
pkg install vm-bhyve
sysrc vm_enable="YES"Next, create a ZFS dataset where your VMs will live. My zpool is named zroot; change if yours is different.
zfs create zroot/vm
sysrc vm_dir="zfs:zroot/vm"Finally, we can initialize things. I assume here that the zpool is mounted at /zroot, and
that your network interface is named em0. This will initialize the bhyve manager, copy
some predefined templates, and set up networking so your VM can talk to the net.
vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/vm/.templates/
vm switch create public
vm switch add public em0I personally like to codify settings in a template because I'm lazy and
forgetful and this probably won't be the first time I have to spin up a VM. You
could skip this step if you like and just edit the resulting config after vm create
but I'll create a template here and talk through the settings.
Here are the contents of mine, which lives at /zroot/vm/.templates/debian-zvol.conf.
loader="grub"
cpu=2
memory=16G
network0_type="virtio-net"
network0_switch="public"
disk0_name="disk0"
disk0_type="virtio-blk"
disk0_dev="sparse-zvol"
grub_run_partition="1"
grub_run_dir="/boot/grub"
Most of the lines are self-explanatory. Obviously, you probably don't need 16G of RAM unless you're doing something crazy with large amounts of data like I am.
The disk settings are worth noting. I've set up a sparse zvol here. I won't elaborate too much on that, but zvols are a cool feature of ZFS. Some people say they aren't as performant as files; the FreeBSD handbook and several others (maybe copying it?) say the opposite. Do your own research. It works well enough and I got it working. You can find some more docs on disks in the vm-bhyve wiki.
Note that there isn't a way to specify the size. I tried a few things and always got a 25G disk. Someone leave a comment if you have a better idea. I ended up doing an ugly dance that we'll come back to in a moment.
Before installing, we need to download an ISO image and let our tool know about it. At the time of this writing, this is the URL for the latest Debian stable network installation image.
vm iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.isoThis is pretty straightforward. We already configured a template above, so we just ask the manager to create a new VM with the template.
vm create -t debian-zvol debguestAt this point, the tool puts some files on disk in /zroot/vm/debguest, most notably
a conf file which looks pretty similar to the template.
Now you can kick off the installer!
vm install debguest debian-11.6.0-amd64-netinst.isoThis tells the manager to start the VM with our installer ISO as the boot device. Now that ugly thing about disks. If you run through the installer you'll realize that the disk has a size of 20G, and I couldn't find a great way to override this (comments welcome).
So after a few seconds I actually power off the vm. The volume does not appear to be created by this tool until the VM is actually powered on. I'm probably holding it wrong. Power it off, set the volume size to what you actually need, then restart the installer.
vm poweroff debguest
zfs set volsize=100G zroot/vm/debguest/disk0
vm install debguest debian-11.6.0-amd64-netinst.isoOk, so what happened? Well, nothing on your terminal. How do you connect to the dang thing?
vm console debguestThis script actually basically just wraps cu, a tool for directly connecting to other machines
from a bygone era, but it actually still works. You might need to hit enter after connecting to wake
it up or something.
Exiting the console is even harder than exiting vi. How on earth to you get out? Well, you have to type
~.. Yes, ~. (or ~ followed by Ctrl-D). And it has to be at the start of a "line" to catch it, which means
you may need to press enter before entering the magic instructions.
That's about it! You now have a VM running Debian, and hopefully figured out how to enter and exit the console.
Check out the manpage for vm to learn more about managing your VM (listing VMs, starting, etc.).
Hope this was helpful!