Skip to content

Instantly share code, notes, and snippets.

@tanb

tanb/xhyve-freebsd-tutorial-1.md

Last active November 27, 2021 13:07
Show Gist options
  • Save tanb/f8fefa22332edc7a641d to your computer and use it in GitHub Desktop.
Save tanb/f8fefa22332edc7a641d to your computer and use it in GitHub Desktop.
Download ZIP
FreeBSD running on xhyve tutorial. (Appendix: Resize image with qemu. Create FreeBSD VM with qemu).
Raw
xhyve-freebsd-tutorial-1.md

TL;DR

  • Create 5GB FreeBSD image.
  • Install FreeBSD on xhyve.
  • Mount host directory.

Requisites

$ brew install xhyve --HEAD

Create Guest Image

This turorial creates 5G image.

$ cd /path/to/workdir
$ mkfile 5g fbsd.img

Install FreeBSD

  • Donwload FreeBSD installer(FreeBSD-10.2-RELEASE-amd64-bootonly.iso) from freebsd.org.
$ ls
FreeBSD-10.2-RELEASE-amd64-bootonly.iso	fbsd.img
  • Create install script (xhyverun-fbsdinstall.sh)
#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="FreeBSD-10.2-RELEASE-amd64-bootonly.iso"
IMG="fbsd.img"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_CD="-s 3:0,ahci-cd,FreeBSD-10.2-RELEASE-amd64-bootonly.iso"
IMG_HDD="-s 4:0,virtio-blk,$IMG"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"
  • Run
$ sudo xhyverun-fbsdinstall.sh

Run The FreeBSD Guest

  • Create run script(xhyverun-fbsd.sh).
#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="fbsd.img"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4:0,virtio-blk,$BOOTVOLUME"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"
  • Run
$ sudo xhyverun-fbsd.sh

Mount Host Directory

On host(192.168.64.1)

$ sudo touch /etc/exports
$ sudo emacs /etc/exports

Edit /etc/exports (exports(5))

/PATH/TO/EXPORTDIR -mapall=501 -network 192.168.64.0 -mask 255.255.255.0

Reread file (nfsd(8))

$ sudo nfsd update

On virtual machine

Mount

# mkdir /usr/home/YOU/host-shared
# sudo mount 192.168.64.1:/PATH/TO/EXPORTDIR /usr/home/YOU/host-shared

Unmount

# mount
/dev/vtbd0p2 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
192.168.64.1:/PATH/TO/EXPORTDIR on /usr/home/YOU/host-shared (nfs)
# sudo umount 192.168.64.1:/PATH/TO/EXPORTDIR
Raw
xhyve-freebsd-tutorial-2.md

TL;DR

  • Resize vm image using qemu-img.

Requisites

$ brew install xhyve --HEAD

$ brew install qemu

Resizing And Growing Disks

Resize VM image size on Host. This example 10GB -> 15GB.

$ qemu-img resize fbsd.raw 15GB

Fix disk partitions configuration on the VM.

# gpart show
=>      34  20971453  vtbd0  GPT  (15G) [CORRUPT]
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930   1048576      3  freebsd-swap  (512M)
  20969506      1981         - free -  (991K)

# gpart recover vtbd0
vtbd0 recovered
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930   1048576      3  freebsd-swap  (512M)
  20969506  10487741         - free -  (5.0G)

Delete swap partition.

# swapoff /dev/vtbd0p3
# gpart delete -i 3 vtbd0
vtbd0p3 deleted
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930  11536317         - free -  (5.5G)

Resize freebsd-ufs partition.

# gpart resize -i 2 -a 4k -s 14G vtbd0
vtbd0p2 resized
root@xhyve-freebsd:~ # gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  29360126      2  freebsd-ufs  (14G)
  29361184   2096063         - free -  (1.0G)

Recreate swap partition.

#  gpart add -t freebsd-swap -a 4k -s 512M vtbd0
vtbd0p3 added
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  29360126      2  freebsd-ufs  (14G)
  29361184   1048576      3  freebsd-swap  (512M)
  30409760   1047487         - free -  (511M)

# swapon /dev/vtbd0p3

Grow the UFS file system.

# growfs /dev/vtbd0p2
Device is mounted read-write; resizing will result in temporary write suspension for /.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/vtbd0p2, mounted on /, from 9.5GB to 14GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
 20516032, 21798272, 23080512, 24362752, 25644992, 26927232, 28209472

Check the resized file system.

# df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/vtbd0p2     14G    1.9G     11G    16%    /
devfs           1.0K    1.0K      0B   100%    /dev
Raw
xhyve-freebsd-tutorial-3.md

TL;DR

  • Create 10GB FreeBSD image using QEMU.
  • Run the VM on xhyve.

Requisites

$ brew install xhyve --HEAD

$ brew install qemu

Create Virtual Machine

  • This Example creates 10GB image.
$ cd /path/to/workdir
$ qemu-img create -f raw fbsd.raw 10G
Formatting 'fbsd.raw', fmt=raw size=10737418240
$ ls
FreeBSD-10.2-RELEASE-amd64-bootonly.iso	fbsd.raw

Install FreeBSD

  • Download FreeBSD installer image(FreeBSD-10.2-RELEASE-amd64-bootonly.iso) from freebsd.org.
$ qemu-system-x86_64 -m 256 -hda fbsd.raw -cdrom FreeBSD-10.2-RELEASE-amd64-bootonly.iso
WARNING: Image format was not specified for 'fbsd.raw' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

QEMU window will open. Install FreeBSD.

Run The VM On Xhyve

#1

Before using xhyve, run the vm on qemu.

$ qemu-system-x86_64 -m 256 -hda fbsd.raw

For VirtIO,

Edit /etc/fstab.

# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/ada0p2    /               ufs     rw      1       1
/dev/ada0p3    none            swap    sw      0       0

To

# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/vtbd0p2    /               ufs     rw      1       1
/dev/vtbd0p3    none            swap    sw      0       0

Edit /etc/rc.conf

ifconfig_vtnet0="DHCP"

Shut it down and close the qemu window.

#2

Create running script(xhyverun-fbsd.sh).

#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="fbsd.raw"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4:0,virtio-blk,$BOOTVOLUME"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"

Run the FreeBSD VM.

$ sudo xhyverun-fbsd.sh

______               ____   _____ _____
|  ____|             |  _ \ / ____|  __ \
| |___ _ __ ___  ___ | |_) | (___ | |  | |
|  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
| |   | | |  __/  __/| |_) |____) | |__| |
| |   | | |    |    ||     |      |      |
|_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                           s` `.....---.......--.```   -/
+============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
|                                         |  yo`:.            :o      `+-

If you got following mount error, you may be forgeting #1.

Trying to mount root from ufs:/dev/ada0p2 [rw]...
mountroot: waiting for device /dev/ada0p2 ...
Mounting from ufs:/dev/ada0p2 failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ada0p2
  vfs.root.mountfrom.options=rw

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/acd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

But you can mount device manually.

mountroot> ?

List of GEOM managed disk devices:
  gptid/5bebeba9-6364-11e5-abe5-bd56998f3bd1 gptid/5be6a257-6364-11e5-abe5-bd56998f3bd1 ufsid/56050e05aac93398 gptid/5be40c9f-6364-11e5-abe5-bd56998f3bd1 diskid/DISK-BHYVE-1C5B-53F3-EA71p3 diskid/DISK-BHYVE-1C5B-53F3-EA71p2 diskid/DISK-BHYVE-1C5B-53F3-EA71p1 vtbd0p3 vtbd0p2 vtbd0p1 diskid/DISK-BHYVE-1C5B-53F3-EA71 vtbd0

mountroot> ufs:ufsid/56050e05aac93398
@zbentley
Copy link

The USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so" lines will not always work. Homebrew might use a different cache, xhyve might not be installed from source, or the source code cache might have been purged, so that file will not exist.

The latest version of the xhyve formula in Homebrew installs these files into the xhyve install dir. You can refer to them properly by updating the USERBOOT lines to read:

USERBOOT="$(brew --prefix xhyve)/share/xhyve/test/userboot.so"

See this SO question for an example of a user affected by this issue.

@vic3t3chn0
Copy link

USERBOOT="$USER/Library/Caches/Homebrew/xhyve--git/test/userboot.so"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment