Skip to content

Instantly share code, notes, and snippets.

@mgerdts
Last active September 12, 2023 16:08
Show Gist options
  • Save mgerdts/6fabc913aca3acd2f1e435a7dc2bbd80 to your computer and use it in GitHub Desktop.
Save mgerdts/6fabc913aca3acd2f1e435a7dc2bbd80 to your computer and use it in GitHub Desktop.
Install windows with bhyve

These are bare-bones instructions for creating a Windows image for bhyve on SmartOS. You will need a platform image that has the fix for OS-7117. Platform images that I've used in testing are here: iso tgz usb

Create a volume that will be the disk.

zfs create -o volblocksize=4k -V 16g -s zones/hdd-windows

This script will be used for booting from the CD.

#!/bin/sh

bootrom=/usr/share/bhyve/uefi-rom.bin

[ "$1" = "-k" ] || pfexec bhyvectl --vm=windows --destroy
bhyve -H -w \
        -s 0,hostbridge \
        -B "1,product=SmartOS HVM" \
        -s 31,lpc \
        -l bootrom,$bootrom \
        -l com1,stdio \
        -c 2 \
        -m 2G \
        -s 3:0,ahci-cd,/zones/media/win2016eval.iso \
        -s 4:0,ahci-hd,/dev/zvol/rdsk/zones/hdd-windows \
        -s 28,fbuf,vga=off,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 29,xhci,tablet \
        windows

You will need to connect via vnc and hit the any key to tell it to boot from the CD. After doing the first chunk of work, Windows willl reboot. You will need to re-run the script one or more times to complete the installation. In the subsequent boots, you do not want to boot from the CD, so don't hit the any key. Note that the ,wait on the fbuf argument will cause bhyve to wait for the VNC connection before booting.

Once the installation is complete, we need to associate zones/hdd-windows with a VM. The simplest way to do that is with the following.

First, stop the windows VM. Shutdown from within the VM or kill the bhyve process (-TERM for an orderly shutdown, -KILL or -9 for pull the cord).

Next, create an empty vm and swap disks.

vmadm create -f bhyve-empty.json
uuid=...
zfs destroy zones/$uuid/disk0
zfs rename zones/hdd-windows zones/$uuid/disk0

Now you should be able to start the Windows VM with vmadm start $uuid

Limitiations

Some things are not fully baked. We're working on them.

  • Currently you must configure networking statically or have an external DHCP server.
  • The procedure above does not include virtio drivers for disks or networking. It is unknown whether any other networking emulation works with bhyve on SmartOS. The virtio drivers we use on kvm and will likely use with bhyve are on this iso. It should be possible to add that iso to the command line during installation and adjust hdd-windows to use virtio-blk instead of ahci.
{
"brand": "bhyve",
"alias": "bhyve-uefi",
"vcpus": 2,
"autoboot": false,
"ram": 2048,
"disks": [
{
"boot": true,
"model": "ahci",
"size": 10240
}
]
}
@GernotS
Copy link

GernotS commented Aug 16, 2018

some questions to this really helpful post:

  • is it enough to configure basic networking in the json just like in KVM and apply a static IP in the host manually?
  • How does one get VNC access after starting the host in a zone?

@plitc
Copy link

plitc commented Dec 24, 2018

@GernotS

open a socat connection for vnc on the global zone:

socat TCP-LISTEN:60001,reuseaddr,fork UNIX-CONNECT:/zones/UUID/root/tmp/vm.vnc

@jubalskaggs
Copy link

Attaching the virtio cd (-s 5:0,ahci-cd,/zones/media/me-ws2012std-20130712.iso \ ) and loading the viostor driver during the first bhyve installation phase works for me with win7 pro - thanks for the instructions! :)

Oddly the mouse doesn't work through the vnc interface during installation, didn't need it to get the installation done.

@limepanda
Copy link

@mgergts I don't see the network device in your command line. How do you get networking? Thanks!

@mike-pt
Copy link

mike-pt commented Sep 29, 2019

-s 28,fbuf,vga=off,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 29,xhci,tablet \

Does this actually work for you? I can't get vnc after starting the bhyve guest, the port is listening but it times out when I try to access via vncclient, I noticed more folks on smartos mailing list saying the same so I'm curious how you got that to work

@mgerdts
Copy link
Author

mgerdts commented Sep 29, 2019

-s 28,fbuf,vga=off,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 29,xhci,tablet \

Does this actually work for you? I can't get vnc after starting the bhyve guest, the port is listening but it times out when I try to access via vncclient, I noticed more folks on smartos mailing list saying the same so I'm curious how you got that to work

The VNC console only works with bootrom=uefi. Even then, the bhyve VNC server implementation is quite limited and does not work well with all clients. I find Chicken of the VNC works well.

If you are trying to have GUI access to a guest for any purpose other than installation, I highly recommend that you use any method other than the bhyve VNC console.

@mike-pt
Copy link

mike-pt commented Sep 29, 2019

Thanks thats what Im using and as for client just the native macOS one.

What other methods exist for GUI install, didn't see anything else in the smartos docs, but then again they are super vague on bhyve

@mike-pt
Copy link

mike-pt commented Sep 29, 2019

ah with that client I see the smartos "bios"/"uefi" loader quickly and than move to black screen so maybe its something wrong with the instalation media

@mike-pt
Copy link

mike-pt commented Sep 29, 2019

btw this is running the bhyve cmd not vmadm yet... I was also using bootrom=/usr/share/bhyve/uefi-csm-rom.bin and switch to bootrom=/usr/share/bhyve/uefi-rom.bin and in this case I get the UEFI shell

@mike-pt
Copy link

mike-pt commented Sep 29, 2019

Tigher vnc seems to work for me 🤷‍♂️ mac native still fails and the other is black or uefi shell (guess It doesn't connect on type to press the key to go fro cd)

@leeyc0
Copy link

leeyc0 commented Mar 11, 2020

Looks like that running bhyve in global zone requires a clean machine that do not have any KVM instances running. I got vm_open: Invalid argument when I run bhyve in a machine with KVM instance without any hint.

I only resolved this problem after I find another machine without any KVM instances.

@matthiasg
Copy link

is there a script or tip on how to convert a kvm to byhve ?

@skirmess
Copy link

What does the -B "1,product=SmartOS HVM" do? Can't find it in the bhyve help.

@mgerdts
Copy link
Author

mgerdts commented Jan 15, 2021

It's been a while (I no longer work on SmartOS), but I believe it sets the BIOS product name, which is visible in a Linux guest with smbios -t 1. I think this was done so that cloud-init will recognize that it should use the SmartOS datasource, but it looks like that is looking for "SmartDC" not "SmartOS".

https://github.com/canonical/cloud-init/blob/0af1ff1eaf593c325b4f53181a572110eb016c50/cloudinit/sources/DataSourceSmartOS.py#L775

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment