pr1ntf/winbhyvesawce.txt

## winbhyvesawce.txt
 Existing Windows installation ISOs expect a GUI console. To get around that, you have to do an "unattended install". Windows people seem to know all about this but it was new to me It basically is dropping down an XML file into the ISO that answers all the questions, and sets parameters.

 Sample XML unattend scripts can be downloaded from
      https://github.com/nahanni/bhyve-windows-unattend-xml

These set up an IP address with DHCP, use "Test123" for the 'Administrator' password, enable remote RDP access, and install virtio drivers. You may want to customize these, but they will work fine out of the box. Note the license keys in these have been scrubbed - they're generally not needed for server installs, but if you're going for a desktop install you'll probably want to add them in.

 Assuming w2k16, the steps are:

- Install the p7zip and cdrtools-devel packages on FreeBSD (the versions I used on CURRENT were v9.38 and 3.01a24,1 respectively)

- Extract the ISO to a working dir. I use 7z for this, though in theory any extractor could work. FreeBSD tar unfortunately won't since the Windows ISOs are UDF.

  # mkdir /unpack
  # cd /unpack
  # 7z x /path/to/win2k16.iso


- Copy in the XML unattend script for win2k16 from the attached tarball, into the root dir of the unpack and rename it

  # cp win2016preview_AutoUnattend.xml /unpack/AutoUnattend.xml

 Note that you can edit some stuff in the XML file if needed - for example, the Administrator password is set to 'Test123' e.g. the fields:

            <UserAccounts>
                <AdministratorPassword>
                    <Value>Test123</Value>

 The license key for 2016 is currently a public one (or at least a non-commercial one) so no problems with that.


- Extract the redhat stable virtio drivers into a virtio subdir of the unpack. This is a stock ISO9660 file so FreeBSD tar can deal with it.

  # fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso
  # mkdir /unpack/virtio
  # cd /unpack/virtio
  # tar xf /path/to/virtio-win-0.1.96.iso .

    (don't forget the dot in the tar command

- Now the ISO can be repacked. The new ISO is built as a hybrid BIOS/UEFI ISO, even though just UEFI in the boot catalog would probably work. The 'noprompt' EFI loader is used since it avoids having to hit return within 5 secs of booting (or EFI exits).

  # cd /unpack

  # mkisofs \
     -b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
     -iso-level 4 -J -l -D \
     -N -joliet-long \
     -relaxed-filenames -v \
     -V "Custom" -udf \
     -boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
     -eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
     -no-emul-boot \
     -o /path/to/output/win2k16_repack.iso .

 The ISO is now readly for the Windows install.


Windows installation
--------------------

Not too much complication here other than the 2 reboots during install.

The trick is to have the CD configured only for the first boot. It has to be removed from the bhyve command line for the second and subsequent boots, since we currently don't have any boot selection in UEFI and a CD is automatically given preference, resulting in an endless cycle of CD installs. For Windows-desktop installs, the ahci-cd drive *must* be kept there, and a 0-byte file used instead e.g. 'touch null.iso'.

There are currently some slot limitations with UEFI:
  - AHCI devices must be in slots 3/4/5/6
  - The PCI-ISA bus aka lpc must be in slot 31
  - virtio-net devices can be in any slot.

You'll want at least a 30G disk image for windows.

A typical command-line would look like:

bhyve \
      -c 1 \
      -s 3,ahci-hd,/images/win2k16.img \
      -s 4,ahci-cd,/images/win2k16_repack.iso \
      -s 10,virtio-net,tap0 \
      -s 31,lpc \
      -l com1,/dev/nmdm0A \
      -l com2,/dev/nmdm1A \
      -l bootrom,/path/to/BHYVE_UEFI.fd \
      -m 1G -H -w \
      w2k16

(Note the UEFI binary file can be grabbed from
   http://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20151002.fd)

The "-A" option isn't used here since UEFI generates it's own ACPI tables.

Another concept to learn is the Windows "SAC" (aka EMS) serial console, which provides a BMC-like interface to Windows. There are a number of built-in commands, and it can also create DOS prompts and switch between them with it's mux. This will be used to monitor the installation process on com1.

Installation slows down with more vCPUs. I'd recommend using 1-2 until the third/final stage, and then firing up the number that you want. I've not done more than 8.

 Another point - 'bhyvectl --destroy' shouldn't be used between invocations of bhyve, unless all resources

Steps
=====

1. First boot: Windows install, stuff copied from the CD to the hard drive. After booting, and about ~15 secs, there will be two additional channels available in the SAC console. These are for output of the autoinstall process:

SAC>
EVENT: The CMD command is now available.
SAC>
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupAct
SAC>
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupErr

SAC>ch
Channel List

(Use "ch -?" for information on using channels)

# Status  Channel Name
0 (AV)    SAC
1 (AV)    SACSetupAct
2 (AV)    SACSetupErr

1a. You can connect to channel 1 and watch stuff fly by. After about 1 minute, the first version of the install on channel1, SACSetupAct, will exit back to SAC, but you can then immediately reconnect for the 2nd phase. This will move reasonably quickly to the main extract:

nPEArchiveOldWindowsFoldersDoneCheckpoint" successfully set.
2015-06-01 17:45:40, Info       [0x06412c] IBSLIB SetCheckpoint: Checkpoint("Wi
nPEImageApplyReadyCheckpoint") in progress...
2015-06-01 17:45:40, Info       [0x06412e] IBSLIB SetCheckpoint: Checkpoint "Wi
nPEImageApplyReadyCheckpoint" successfully set.
2015-06-01 17:45:40, Info       [0x06009e] IBS    DeployWIMImage:Calling IDepWI
MImageResolved::Apply...
2015-06-01 17:45:40, Info       [0x0606cc] IBS    Calling WIMApplyImage (flags
= 0x180)...

 .. and will set here for about ~5 mins, using up lots of CPU as it copies stuff from the ISO to the hard drive, very visible with 'top -H' on the host.

 It then starts installing the virtio drivers which takes a few minutes (could be reduced by pruning out unnecessary Windows versions and driver types when doing the extract above). After this is done, the VM will shut done, ready for the 2nd phase.

 This phase takes about ~8 mins on my IvyBridge.


2. Second boot, from the hard drive, for SYSPREP.

 For this next step, the CD has to be taken out from the command line so Windows can be booted from the hard drive. The SACSetupAct and SACSetupErr channels are available in this phase, but take a good ~25 seconds to come up. However, the SYSPREP phase is much quicker.

 bhyve will spit out some errors for unimplemented SATA commands but these can be ignored:

Unsupported cmd:5d
Unsupported cmd:5d
Unsupported cmd:f5


 There is a progress indicator for the install on SACSetupAct channel:

plete = 2, Progress = 33% (33%)
2015-06-01 18:16:36, Info                  SYSPRP SPPNP: Status: Active
2015-06-01 18:16:36, Info                  SYSPRP SPPNP: Status: Total = 6, Com
plete = 2, Progress = 33% (33%)


 When it's done, Windows reboots and the VM exits. This phase took 2min30sec for me.


3. Third boot, Windows proper, first time.

 Also without the CD configured.

 This boot is a little slow. It takes ~40 secs for CMD to be available, and gobs of CPU are used for quite a long time (~15 mins ?). It appears that Windows does a whole lot of work during this time, but it eventually does settle down. Still not perfect: server 2016 appears to use ~2% host CPU when idle.

 Shutdown is also a little slow on the first real boot.

 The virtio interface should use DHCP by default, but you can override that using the 'i' command from the SAC console or your favourite command-line net tool. The XML auto-install script enables RDP with just password auth, and allows RDP through the firewall so you should be able to connect via RDP to the VM once it's up.

 To find out the IP address if DHCP is used, the 'i' command in SAC will display it.

 Note that RDP isn't available for quite a few minutes on the first boot. You'll see rejected connections, but it will eventually let you in. Even then, the server manager takes a few more minutes to show up.

 I'm using the Microsoft Remote Desktop client, v8.0.16, on OSX. That seems to work pretty well. We've also tried FreeRDP, which seems to work great but doesn't support dynamic window sizing.

 Let me know if you need any help/clarifications.
	Existing Windows installation ISOs expect a GUI console. To get around that, you have to do an "unattended install". Windows people seem to know all about this but it was new to me It basically is dropping down an XML file into the ISO that answers all the questions, and sets parameters.

	Sample XML unattend scripts can be downloaded from
	https://github.com/nahanni/bhyve-windows-unattend-xml

	These set up an IP address with DHCP, use "Test123" for the 'Administrator' password, enable remote RDP access, and install virtio drivers. You may want to customize these, but they will work fine out of the box. Note the license keys in these have been scrubbed - they're generally not needed for server installs, but if you're going for a desktop install you'll probably want to add them in.

	Assuming w2k16, the steps are:

	- Install the p7zip and cdrtools-devel packages on FreeBSD (the versions I used on CURRENT were v9.38 and 3.01a24,1 respectively)

	- Extract the ISO to a working dir. I use 7z for this, though in theory any extractor could work. FreeBSD tar unfortunately won't since the Windows ISOs are UDF.

	# mkdir /unpack
	# cd /unpack
	# 7z x /path/to/win2k16.iso


	- Copy in the XML unattend script for win2k16 from the attached tarball, into the root dir of the unpack and rename it

	# cp win2016preview_AutoUnattend.xml /unpack/AutoUnattend.xml

	Note that you can edit some stuff in the XML file if needed - for example, the Administrator password is set to 'Test123' e.g. the fields:

	<UserAccounts>
	<AdministratorPassword>
	<Value>Test123</Value>

	The license key for 2016 is currently a public one (or at least a non-commercial one) so no problems with that.


	- Extract the redhat stable virtio drivers into a virtio subdir of the unpack. This is a stock ISO9660 file so FreeBSD tar can deal with it.

	# fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso
	# mkdir /unpack/virtio
	# cd /unpack/virtio
	# tar xf /path/to/virtio-win-0.1.96.iso .

	(don't forget the dot in the tar command

	- Now the ISO can be repacked. The new ISO is built as a hybrid BIOS/UEFI ISO, even though just UEFI in the boot catalog would probably work. The 'noprompt' EFI loader is used since it avoids having to hit return within 5 secs of booting (or EFI exits).

	# cd /unpack

	# mkisofs \
	-b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
	-iso-level 4 -J -l -D \
	-N -joliet-long \
	-relaxed-filenames -v \
	-V "Custom" -udf \
	-boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
	-eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
	-no-emul-boot \
	-o /path/to/output/win2k16_repack.iso .

	The ISO is now readly for the Windows install.


	Windows installation
	--------------------

	Not too much complication here other than the 2 reboots during install.

	The trick is to have the CD configured only for the first boot. It has to be removed from the bhyve command line for the second and subsequent boots, since we currently don't have any boot selection in UEFI and a CD is automatically given preference, resulting in an endless cycle of CD installs. For Windows-desktop installs, the ahci-cd drive must be kept there, and a 0-byte file used instead e.g. 'touch null.iso'.

	There are currently some slot limitations with UEFI:
	- AHCI devices must be in slots 3/4/5/6
	- The PCI-ISA bus aka lpc must be in slot 31
	- virtio-net devices can be in any slot.

	You'll want at least a 30G disk image for windows.

	A typical command-line would look like:

	bhyve \
	-c 1 \
	-s 3,ahci-hd,/images/win2k16.img \
	-s 4,ahci-cd,/images/win2k16_repack.iso \
	-s 10,virtio-net,tap0 \
	-s 31,lpc \
	-l com1,/dev/nmdm0A \
	-l com2,/dev/nmdm1A \
	-l bootrom,/path/to/BHYVE_UEFI.fd \
	-m 1G -H -w \
	w2k16

	(Note the UEFI binary file can be grabbed from
	http://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20151002.fd)

	The "-A" option isn't used here since UEFI generates it's own ACPI tables.

	Another concept to learn is the Windows "SAC" (aka EMS) serial console, which provides a BMC-like interface to Windows. There are a number of built-in commands, and it can also create DOS prompts and switch between them with it's mux. This will be used to monitor the installation process on com1.

	Installation slows down with more vCPUs. I'd recommend using 1-2 until the third/final stage, and then firing up the number that you want. I've not done more than 8.

	Another point - 'bhyvectl --destroy' shouldn't be used between invocations of bhyve, unless all resources

	Steps
	=====

	1. First boot: Windows install, stuff copied from the CD to the hard drive. After booting, and about ~15 secs, there will be two additional channels available in the SAC console. These are for output of the autoinstall process:

	SAC>
	EVENT: The CMD command is now available.
	SAC>
	EVENT: A new channel has been created. Use "ch -?" for channel help.
	Channel: SACSetupAct
	SAC>
	EVENT: A new channel has been created. Use "ch -?" for channel help.
	Channel: SACSetupErr

	SAC>ch
	Channel List

	(Use "ch -?" for information on using channels)

	# Status Channel Name
	0 (AV) SAC
	1 (AV) SACSetupAct
	2 (AV) SACSetupErr

	1a. You can connect to channel 1 and watch stuff fly by. After about 1 minute, the first version of the install on channel1, SACSetupAct, will exit back to SAC, but you can then immediately reconnect for the 2nd phase. This will move reasonably quickly to the main extract:

	nPEArchiveOldWindowsFoldersDoneCheckpoint" successfully set.
	2015-06-01 17:45:40, Info [0x06412c] IBSLIB SetCheckpoint: Checkpoint("Wi
	nPEImageApplyReadyCheckpoint") in progress...
	2015-06-01 17:45:40, Info [0x06412e] IBSLIB SetCheckpoint: Checkpoint "Wi
	nPEImageApplyReadyCheckpoint" successfully set.
	2015-06-01 17:45:40, Info [0x06009e] IBS DeployWIMImage:Calling IDepWI
	MImageResolved::Apply...
	2015-06-01 17:45:40, Info [0x0606cc] IBS Calling WIMApplyImage (flags
	= 0x180)...

	.. and will set here for about ~5 mins, using up lots of CPU as it copies stuff from the ISO to the hard drive, very visible with 'top -H' on the host.

	It then starts installing the virtio drivers which takes a few minutes (could be reduced by pruning out unnecessary Windows versions and driver types when doing the extract above). After this is done, the VM will shut done, ready for the 2nd phase.

	This phase takes about ~8 mins on my IvyBridge.


	2. Second boot, from the hard drive, for SYSPREP.

	For this next step, the CD has to be taken out from the command line so Windows can be booted from the hard drive. The SACSetupAct and SACSetupErr channels are available in this phase, but take a good ~25 seconds to come up. However, the SYSPREP phase is much quicker.

	bhyve will spit out some errors for unimplemented SATA commands but these can be ignored:

	Unsupported cmd:5d
	Unsupported cmd:5d
	Unsupported cmd:f5


	There is a progress indicator for the install on SACSetupAct channel:

	plete = 2, Progress = 33% (33%)
	2015-06-01 18:16:36, Info SYSPRP SPPNP: Status: Active
	2015-06-01 18:16:36, Info SYSPRP SPPNP: Status: Total = 6, Com
	plete = 2, Progress = 33% (33%)


	When it's done, Windows reboots and the VM exits. This phase took 2min30sec for me.


	3. Third boot, Windows proper, first time.

	Also without the CD configured.

	This boot is a little slow. It takes ~40 secs for CMD to be available, and gobs of CPU are used for quite a long time (~15 mins ?). It appears that Windows does a whole lot of work during this time, but it eventually does settle down. Still not perfect: server 2016 appears to use ~2% host CPU when idle.

	Shutdown is also a little slow on the first real boot.

	The virtio interface should use DHCP by default, but you can override that using the 'i' command from the SAC console or your favourite command-line net tool. The XML auto-install script enables RDP with just password auth, and allows RDP through the firewall so you should be able to connect via RDP to the VM once it's up.

	To find out the IP address if DHCP is used, the 'i' command in SAC will display it.

	Note that RDP isn't available for quite a few minutes on the first boot. You'll see rejected connections, but it will eventually let you in. Even then, the server manager takes a few more minutes to show up.

	I'm using the Microsoft Remote Desktop client, v8.0.16, on OSX. That seems to work pretty well. We've also tried FreeRDP, which seems to work great but doesn't support dynamic window sizing.

	Let me know if you need any help/clarifications.