Skip to content

Instantly share code, notes, and snippets.

@tanb

tanb/xhyve-freebsd-tutorial-1.md

Last active November 27, 2021 13:07
Show Gist options
  • Star 80 You must be signed in to star a gist
  • Fork 19 You must be signed in to fork a gist
  • Save tanb/f8fefa22332edc7a641d to your computer and use it in GitHub Desktop.
Save tanb/f8fefa22332edc7a641d to your computer and use it in GitHub Desktop.
Download ZIP
FreeBSD running on xhyve tutorial. (Appendix: Resize image with qemu. Create FreeBSD VM with qemu).
Raw
xhyve-freebsd-tutorial-1.md

TL;DR

  • Create 5GB FreeBSD image.
  • Install FreeBSD on xhyve.
  • Mount host directory.

Requisites

$ brew install xhyve --HEAD

Create Guest Image

This turorial creates 5G image.

$ cd /path/to/workdir
$ mkfile 5g fbsd.img

Install FreeBSD

  • Donwload FreeBSD installer(FreeBSD-10.2-RELEASE-amd64-bootonly.iso) from freebsd.org.
$ ls
FreeBSD-10.2-RELEASE-amd64-bootonly.iso	fbsd.img
  • Create install script (xhyverun-fbsdinstall.sh)
#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="FreeBSD-10.2-RELEASE-amd64-bootonly.iso"
IMG="fbsd.img"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_CD="-s 3:0,ahci-cd,FreeBSD-10.2-RELEASE-amd64-bootonly.iso"
IMG_HDD="-s 4:0,virtio-blk,$IMG"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"
  • Run
$ sudo xhyverun-fbsdinstall.sh

Run The FreeBSD Guest

  • Create run script(xhyverun-fbsd.sh).
#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="fbsd.img"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4:0,virtio-blk,$BOOTVOLUME"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"
  • Run
$ sudo xhyverun-fbsd.sh

Mount Host Directory

On host(192.168.64.1)

$ sudo touch /etc/exports
$ sudo emacs /etc/exports

Edit /etc/exports (exports(5))

/PATH/TO/EXPORTDIR -mapall=501 -network 192.168.64.0 -mask 255.255.255.0

Reread file (nfsd(8))

$ sudo nfsd update

On virtual machine

Mount

# mkdir /usr/home/YOU/host-shared
# sudo mount 192.168.64.1:/PATH/TO/EXPORTDIR /usr/home/YOU/host-shared

Unmount

# mount
/dev/vtbd0p2 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
192.168.64.1:/PATH/TO/EXPORTDIR on /usr/home/YOU/host-shared (nfs)
# sudo umount 192.168.64.1:/PATH/TO/EXPORTDIR
Raw
xhyve-freebsd-tutorial-2.md

TL;DR

  • Resize vm image using qemu-img.

Requisites

$ brew install xhyve --HEAD

$ brew install qemu

Resizing And Growing Disks

Resize VM image size on Host. This example 10GB -> 15GB.

$ qemu-img resize fbsd.raw 15GB

Fix disk partitions configuration on the VM.

# gpart show
=>      34  20971453  vtbd0  GPT  (15G) [CORRUPT]
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930   1048576      3  freebsd-swap  (512M)
  20969506      1981         - free -  (991K)

# gpart recover vtbd0
vtbd0 recovered
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930   1048576      3  freebsd-swap  (512M)
  20969506  10487741         - free -  (5.0G)

Delete swap partition.

# swapoff /dev/vtbd0p3
# gpart delete -i 3 vtbd0
vtbd0p3 deleted
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  19919872      2  freebsd-ufs  (9.5G)
  19920930  11536317         - free -  (5.5G)

Resize freebsd-ufs partition.

# gpart resize -i 2 -a 4k -s 14G vtbd0
vtbd0p2 resized
root@xhyve-freebsd:~ # gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  29360126      2  freebsd-ufs  (14G)
  29361184   2096063         - free -  (1.0G)

Recreate swap partition.

#  gpart add -t freebsd-swap -a 4k -s 512M vtbd0
vtbd0p3 added
# gpart show
=>      34  31457213  vtbd0  GPT  (15G)
        34      1024      1  freebsd-boot  (512K)
      1058  29360126      2  freebsd-ufs  (14G)
  29361184   1048576      3  freebsd-swap  (512M)
  30409760   1047487         - free -  (511M)

# swapon /dev/vtbd0p3

Grow the UFS file system.

# growfs /dev/vtbd0p2
Device is mounted read-write; resizing will result in temporary write suspension for /.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/vtbd0p2, mounted on /, from 9.5GB to 14GB? [Yes/No] Yes
super-block backups (for fsck_ffs -b #) at:
 20516032, 21798272, 23080512, 24362752, 25644992, 26927232, 28209472

Check the resized file system.

# df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/vtbd0p2     14G    1.9G     11G    16%    /
devfs           1.0K    1.0K      0B   100%    /dev
Raw
xhyve-freebsd-tutorial-3.md

TL;DR

  • Create 10GB FreeBSD image using QEMU.
  • Run the VM on xhyve.

Requisites

$ brew install xhyve --HEAD

$ brew install qemu

Create Virtual Machine

  • This Example creates 10GB image.
$ cd /path/to/workdir
$ qemu-img create -f raw fbsd.raw 10G
Formatting 'fbsd.raw', fmt=raw size=10737418240
$ ls
FreeBSD-10.2-RELEASE-amd64-bootonly.iso	fbsd.raw

Install FreeBSD

  • Download FreeBSD installer image(FreeBSD-10.2-RELEASE-amd64-bootonly.iso) from freebsd.org.
$ qemu-system-x86_64 -m 256 -hda fbsd.raw -cdrom FreeBSD-10.2-RELEASE-amd64-bootonly.iso
WARNING: Image format was not specified for 'fbsd.raw' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

QEMU window will open. Install FreeBSD.

Run The VM On Xhyve

#1

Before using xhyve, run the vm on qemu.

$ qemu-system-x86_64 -m 256 -hda fbsd.raw

For VirtIO,

Edit /etc/fstab.

# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/ada0p2    /               ufs     rw      1       1
/dev/ada0p3    none            swap    sw      0       0

To

# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/vtbd0p2    /               ufs     rw      1       1
/dev/vtbd0p3    none            swap    sw      0       0

Edit /etc/rc.conf

ifconfig_vtnet0="DHCP"

Shut it down and close the qemu window.

#2

Create running script(xhyverun-fbsd.sh).

#!/bin/sh

UUID="-U deaddead-dead-dead-dead-deaddeaddead"

USERBOOT="/Library/Caches/Homebrew/xhyve--git/test/userboot.so"
BOOTVOLUME="fbsd.raw"
KERNELENV=""

MEM="-m 2G"
SMP="-c 2"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4:0,virtio-blk,$BOOTVOLUME"
LPC_DEV="-l com1,stdio"
ACPI="-A"

xhyve $ACPI $MEM $SMP $PCI_DEV $LPC_DEV $NET $IMG_HDD $UUID -f fbsd,$USERBOOT,$BOOTVOLUME,"$KERNELENV"

Run the FreeBSD VM.

$ sudo xhyverun-fbsd.sh

______               ____   _____ _____
|  ____|             |  _ \ / ____|  __ \
| |___ _ __ ___  ___ | |_) | (___ | |  | |
|  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
| |   | | |  __/  __/| |_) |____) | |__| |
| |   | | |    |    ||     |      |      |
|_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                           s` `.....---.......--.```   -/
+============Welcome to FreeBSD===========+ +o   .--`         /y:`      +.
|                                         |  yo`:.            :o      `+-

If you got following mount error, you may be forgeting #1.

Trying to mount root from ufs:/dev/ada0p2 [rw]...
mountroot: waiting for device /dev/ada0p2 ...
Mounting from ufs:/dev/ada0p2 failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ada0p2
  vfs.root.mountfrom.options=rw

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:tank
        cd9660:/dev/acd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

But you can mount device manually.

mountroot> ?

List of GEOM managed disk devices:
  gptid/5bebeba9-6364-11e5-abe5-bd56998f3bd1 gptid/5be6a257-6364-11e5-abe5-bd56998f3bd1 ufsid/56050e05aac93398 gptid/5be40c9f-6364-11e5-abe5-bd56998f3bd1 diskid/DISK-BHYVE-1C5B-53F3-EA71p3 diskid/DISK-BHYVE-1C5B-53F3-EA71p2 diskid/DISK-BHYVE-1C5B-53F3-EA71p1 vtbd0p3 vtbd0p2 vtbd0p1 diskid/DISK-BHYVE-1C5B-53F3-EA71 vtbd0

mountroot> ufs:ufsid/56050e05aac93398
@vic3t3chn0
Copy link

USERBOOT="$USER/Library/Caches/Homebrew/xhyve--git/test/userboot.so"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment