High vCPU Counts in bhyve

0-README.txt

Below are the outputs of the host dmesg.boot, the changes made to vmm.h, the guest demsg.boot at 16 vCPUs, and the guest console output when trying to boot with 26 vCPUs. 
Booting up to 25 vCPUs will work just fine. 

I also included the script used to boot the bhyve instance. 

1-host-vmm-edit.txt

[redacted]:~/highvcpu % cat /usr/src/sys/amd64/include/vmm.h | grep VM_MAXCPU
#define VM_MAXCPU       32                      /* maximum virtual cpus */

2-host-dmesg.boot.txt

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r305758M: Tue Sep 13 07:51:17 UTC 2016
    root@[redacted]:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): resolution 640x480
Cannot accommodate vmm stat type "ipis sent to vcpu"!
CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (2500.12-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 66627448832 (63540 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 40 CPUs
FreeBSD/SMP: 2 package(s) x 10 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
random: entropy device external interface
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f3be30, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 0 on acpi0
cpu2: <ACPI CPU> numa-domain 0 on acpi0
cpu3: <ACPI CPU> numa-domain 0 on acpi0
cpu4: <ACPI CPU> numa-domain 0 on acpi0
cpu5: <ACPI CPU> numa-domain 0 on acpi0
cpu6: <ACPI CPU> numa-domain 0 on acpi0
cpu7: <ACPI CPU> numa-domain 0 on acpi0
cpu8: <ACPI CPU> numa-domain 0 on acpi0
cpu9: <ACPI CPU> numa-domain 0 on acpi0
cpu10: <ACPI CPU> numa-domain 0 on acpi0
cpu11: <ACPI CPU> numa-domain 0 on acpi0
cpu12: <ACPI CPU> numa-domain 0 on acpi0
cpu13: <ACPI CPU> numa-domain 0 on acpi0
cpu14: <ACPI CPU> numa-domain 0 on acpi0
cpu15: <ACPI CPU> numa-domain 0 on acpi0
cpu16: <ACPI CPU> numa-domain 0 on acpi0
cpu17: <ACPI CPU> numa-domain 0 on acpi0
cpu18: <ACPI CPU> numa-domain 0 on acpi0
cpu19: <ACPI CPU> numa-domain 0 on acpi0
cpu20: <ACPI CPU> numa-domain 1 on acpi0
cpu21: <ACPI CPU> numa-domain 1 on acpi0
cpu22: <ACPI CPU> numa-domain 1 on acpi0
cpu23: <ACPI CPU> numa-domain 1 on acpi0
cpu24: <ACPI CPU> numa-domain 1 on acpi0
cpu25: <ACPI CPU> numa-domain 1 on acpi0
cpu26: <ACPI CPU> numa-domain 1 on acpi0
cpu27: <ACPI CPU> numa-domain 1 on acpi0
cpu28: <ACPI CPU> numa-domain 1 on acpi0
cpu29: <ACPI CPU> numa-domain 1 on acpi0
cpu30: <ACPI CPU> numa-domain 1 on acpi0
cpu31: <ACPI CPU> numa-domain 1 on acpi0
cpu32: <ACPI CPU> numa-domain 1 on acpi0
cpu33: <ACPI CPU> numa-domain 1 on acpi0
cpu34: <ACPI CPU> numa-domain 1 on acpi0
cpu35: <ACPI CPU> numa-domain 1 on acpi0
cpu36: <ACPI CPU> numa-domain 1 on acpi0
cpu37: <ACPI CPU> numa-domain 1 on acpi0
cpu38: <ACPI CPU> numa-domain 1 on acpi0
cpu39: <ACPI CPU> numa-domain 1 on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 26 at device 1.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 06.709.07.00-fbsd
mfi0: <ThunderBolt> port 0x8000-0x80ff mem 0xdfb60000-0xdfb63fff,0xdfb00000-0xdfb3ffff irq 16 at device 0.0 numa-domain 0 on pci1
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23 
mfi0: FW MaxCmds = 1008, limiting to 128
mfi0: MaxCmd = 1008, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c03f0
pcib2: <ACPI PCI-PCI bridge> irq 32 at device 2.0 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 32 at device 2.2 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 40 at device 3.0 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 40 at device 3.2 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x7020-0x703f mem 0xdfa20000-0xdfa3ffff,0xdfac4000-0xdfac7fff irq 42 at device 0.0 numa-domain 0 on pci5
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: xxxx
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb0: netmap queues/slots: TX 8/1024, RX 8/1024
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x7000-0x701f mem 0xdfa00000-0xdfa1ffff,0xdfac0000-0xdfac3fff irq 45 at device 0.1 numa-domain 0 on pci5
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: xxxx
igb1: Bound queue 0 to cpu 8
igb1: Bound queue 1 to cpu 9
igb1: Bound queue 2 to cpu 10
igb1: Bound queue 3 to cpu 11
igb1: Bound queue 4 to cpu 12
igb1: Bound queue 5 to cpu 13
igb1: Bound queue 6 to cpu 14
igb1: Bound queue 7 to cpu 15
igb1: netmap queues/slots: TX 8/1024, RX 8/1024
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 17.0 numa-domain 0 on pci0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
isci0: <Intel(R) C600 Series Chipset SAS Controller (SATA mode)> port 0x6000-0x60ff mem 0xde47c000-0xde47ffff,0xde000000-0xde3fffff irq 16 at device 0.0 numa-domain 0 on pci6
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xdfc24000-0xdfc243ff irq 16 at device 26.0 numa-domain 0 on pci0
usbus0: EHCI version 1.0
usbus0 numa-domain 0 on ehci0
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xdfc23000-0xdfc233ff irq 23 at device 29.0 numa-domain 0 on pci0
usbus1: EHCI version 1.0
usbus1 numa-domain 0 on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
vgapci0: <VGA-compatible display> mem 0xdd000000-0xddffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 16 at device 3.0 numa-domain 0 on pci7
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
ahci0: <Intel Patsburg AHCI SATA controller> port 0x9070-0x9077,0x9060-0x9063,0x9050-0x9057,0x9040-0x9043,0x9020-0x903f mem 0xdfc22000-0xdfc227ff irq 19 at device 31.2 numa-domain 0 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib8: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib9: _OSC returned error 0x10
pci9: <ACPI PCI bus> numa-domain 1 on pcib9
pcib10: <ACPI PCI-PCI bridge> irq 50 at device 1.0 numa-domain 1 on pci9
pci10: <ACPI PCI bus> numa-domain 1 on pcib10
pcib11: <ACPI PCI-PCI bridge> irq 56 at device 2.0 numa-domain 1 on pci9
pci11: <ACPI PCI bus> numa-domain 1 on pcib11
pcib12: <ACPI PCI-PCI bridge> irq 56 at device 2.2 numa-domain 1 on pci9
pci12: <ACPI PCI bus> numa-domain 1 on pcib12
pcib13: <ACPI PCI-PCI bridge> irq 64 at device 3.0 numa-domain 1 on pci9
pci13: <ACPI PCI bus> numa-domain 1 on pcib13
pcib14: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pci14: <ACPI PCI bus> numa-domain 1 on pcib14
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcf7ff,0xcf800-0xd07ff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
est16: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu16
est17: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu17
est18: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu18
est19: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu19
est20: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu20
est21: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu21
est22: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu22
est23: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu23
est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
est32: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu32
est33: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu33
est34: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu34
est35: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu35
est36: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu36
est37: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu37
est38: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu38
est39: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu39
mfi0: 5976 (527127550s/0x0020/info) - Shutdown command received from host
mfi0: 5977 (boot + 8s/0x0020/info) - Firmware initialization started (PCI ID 005b/1000/0690/15d9)
mfi0: 5978 (boot + 8s/0x0020/info) - Firmware version 3.400.05-3175
mfi0: 5979 (boot + 15s/0x0020/info) - Package version 23.28.0-0010
mfi0: 5980 (boot + 15s/0x0020/info) - Board Revision 
mfi0: 5981 (boot + 36s/0x0004/info) - Enclosure (SES) discovered on PD 00(c Port 0 - 3/p1)
mfi0: 5982 (boot + 36s/0x0004/info) - Enclosure PD 00(c Port 0 - 3/p1) communication restored
mfi0: 5983 (boot + 36s/0x0004/CRIT) - Enclosure PD 00(c Port 0 - 3/p1) fan 1 failed
mfi0: 5984 (boot + 36s/0x0004/CRIT) - Enclosure PD 00(c Port 0 - 3/p1) fan 2 failed
mfi0: 5985 (boot + 36s/0x0004/CRIT) - Enclosure PD 00(c Port 0 - 3/p1) fan 3 failed
mfi0: 5986 (boot + 36s/0x0004/WARN) - Enclosure PD 00(c Port 0 - 3/p1) power supply 1 removed
mfi0: 5987 (boot + 36s/0x0004/WARN) - Enclosure PD 00(c Port 0 - 3/p1) power supply 2 removed
mfi0: 5988 (boot + 36s/0x0002/info) - Inserted: Encl PD 00
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
mfisyspd0 numa-domain 0 on mfi0
mfisyspd0: 171705MB (351651888 sectors) SYSPD volume (deviceid: 1)
mfisyspd0:  SYSPD volume attached
mfisyspd1 numa-domain 0 on mfi0
mfisyspd1: 171705MB (351651888 sectors) SYSPD volume (deviceid: 2)
mfisyspd1:  SYSPD volume attached
mfisyspd2 numa-domain 0 on mfi0
mfisyspd2: 476940MB (976773168 sectors) SYSPD volume (deviceid: 3)
mfisyspd2:  SYSPD volume attached
mfisyspd3 numa-domain 0 on mfi0
mfisyspd3: 476940MB (976773168 sectors) SYSPD volume (deviceid: 4)
mfisyspd3:  SYSPD volume attached
mfisyspd4 numa-domain 0 on mfi0
mfisyspd4: 476940MB (976773168 sectors) SYSPD volume (deviceid: 5)
mfisyspd4:  SYSPD volume attached
mfisyspd5 numa-domain 0 on mfi0
mfisyspd5: 476940MB (976773168 sectors) SYSPD volume (deviceid: 6)
mfisyspd5:  SYSPD volume attached
mfi0: 5989 (boot + 36s/0x0002/info) - Inserted: PD 00(c Port 0 - 3/p1) Info: enclPd=00, scsiType=d, portMap=00, sasAddr=500304800174fe7d,0000000000000000
mfi0: 5990 (boot + 36s/0x0002/info) - Inserted: PD 01(e0x00/s0)
mfi0: 5991 (boot + 36s/0x0002/info) - Inserted: PD 01(e0x00/s0) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe6c,0000000000000000
mfi0: 5992 (boot + 36s/0x0002/info) - Inserted: PD 02(e0x00/s1)
mfi0: 5993 (boot + 36s/0x0002/info) - Inserted: PD 02(e0x00/s1) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe6d,0000000000000000
mfi0: 5994 (boot + 36s/0x0002/info) - Inserted: PD 03(e0x00/s3)
mfi0: 5995 (boot + 36s/0x0002/info) - Inserted: PD 03(e0x00/s3) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe6f,0000000000000000
mfi0: 5996 (boot + 36s/0x0002/info) - Inserted: PD 04(e0x00/s5)
mfi0: 5997 (boot + 36s/0x0002/info) - Inserted: PD 04(e0x00/s5) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe71,0000000000000000
mfi0: 5998 (boot + 36s/0x0002/info) - Inserted: PD 05(e0x00/s4)
mfi0: 5999 (boot + 36s/0x0002/info) - Inserted: PD 05(e0x00/s4) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe70,0000000000000000
mfi0: 6000 (boot + 36s/0x0002/info) - Inserted: PD 06(e0x00/s2)
mfi0: 6001 (boot + 36s/0x0002/info) - Inserted: PD 06(e0x00/s2) Info: enclPd=00, scsiType=0, portMap=00, sasAddr=500304800174fe6e,0000000000000000
mfi0: 6002 (527127591s/0x0020/info) - Time established as 09/14/16  0:19:51; (42 seconds since power on)
mfi0: 6003 (527127665s/0x0020/info) - Host driver is loaded and operational
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
SMP: AP CPU #24 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #21 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #26 Launched!
Timecounter "TSC-low" frequency 1250062392 Hz quality 1000
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from zfs:zroot/ROOT/default []...
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2 numa-domain 0 on uhub0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3 numa-domain 0 on uhub1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ukbd0 numa-domain 0 on uhub2
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
bridge0: Ethernet address: 02:13:84:e4:f8:00
tap0: Ethernet address: 00:bd:4c:2a:f7:00
igb0: promiscuous mode enabled
bridge0: link state changed to UP
tap0: promiscuous mode enabled
bridge0: link state changed to DOWN
ums0 numa-domain 0 on uhub2
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [XYZ] coordinates ID=0
igb0: link state changed to UP
bridge0: link state changed to UP

3-guest-dmesg.boot.txt

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r305028: Mon Aug 29 22:45:29 UTC 2016
    root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (2499.74-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
  Features2=0xfe9e6217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x200<ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory  = 35433480192 (33792 MB)
avail memory = 33280393216 (31738 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVMADT  >
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 16 package(s)
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-31 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f39d70, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <BHYVE BVXSDT> on motherboard
acpi0: Power Button (fixed)
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 10000000 Hz quality 950
Event timer "HPET" frequency 10000000 Hz quality 350
Event timer "HPET1" frequency 10000000 Hz quality 350
Event timer "HPET2" frequency 10000000 Hz quality 350
Event timer "HPET3" frequency 10000000 Hz quality 350
Event timer "HPET4" frequency 10000000 Hz quality 350
Event timer "HPET5" frequency 10000000 Hz quality 350
Event timer "HPET6" frequency 10000000 Hz quality 350
Event timer "HPET7" frequency 10000000 Hz quality 350
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
ahci0: <Intel ICH8 AHCI SATA controller> mem 0xc0000000-0xc00003ff irq 16 at device 2.0 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahci1: <Intel ICH8 AHCI SATA controller> mem 0xc0000400-0xc00007ff irq 17 at device 4.0 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich6: <AHCI channel> at channel 0 on ahci1
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0002000-0xc0003fff irq 18 at device 8.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:a0:98:93:7f:51
000.000049 [ 421] vtnet_netmap_attach       max rings 1
vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024
000.000050 [ 426] vtnet_netmap_attach       virtio attached txq=1, txd=1024 rxq=1, rxd=1024
isab0: <PCI-ISA bridge> at device 16.0 on pci0
isa0: <ISA bus> on isab0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
driver bug: Unable to set devclass (class: atkbdc devname: (unknown))
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada0: Serial Number BHYVE-9AE8-BCF5-0AB9
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 16384MB (33554432 512 byte sectors)
ada1 at ahcich6 bus 0 scbus1 target 0 lun 0
ada1: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada1: Serial Number BHYVE-72C7-0D38-58BC
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 684MB (1401746 512 byte sectors)
SMP: AP CPU #10 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #8 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ada0s1a [rw]...
vtnet0: link state changed to UP

4-guest-26vcpucrash.txt

GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r305028: Mon Aug 29 22:45:29 UTC 2016
    root@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (2499.74-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
  
Features2=0xfe9e6217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,H
V>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x200<ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory  = 35433480192 (33792 MB)
avail memory = 33280393216 (31738 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVMADT  >
FreeBSD/SMP: Multiprocessor System Detected: 26 CPUs
FreeBSD/SMP: 26 package(s)
random: unblocking device.
ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, should be 0x0A (20160527/tbprint-229)                      
[7/1843]
MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f39d70, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <BHYVE BVXSDT> on motherboard
acpi0: could not allocate interrupt
ACPI Exception: AE_ALREADY_EXISTS, Unable to install System Control Interrupt handler (20160527/evevent-152)
acpi0: Could not enable ACPI: AE_ALREADY_EXISTS
device_attach: acpi0 attach returned 6


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x60
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80ff3859
stack pointer           = 0x28:0xffffffff821c19d0
frame pointer           = 0x28:0xffffffff821c1a50
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (swapper)
[ thread pid 0 tid 100000 ]
Stopped at      acpi_install_wakeup_handler+0x1d9:      movq    %rdi,0x60(%r14)
db> 

5-yabs.sh

#!/bin/sh

# Yet Another bhyve Script v0.2
# High CPU Testing

name=overkill
ram=32768M
cpu=26
installer=FreeBSD-12.0-CURRENT-amd64-20160829-r305028-memstick.img
disk=overkill.img
tap=tap0
con=stdio
fw=BHYVE_UEFI_20151002.fd

#       -l bootrom,${fw} \
#       -s 4,ahci-hd,${installer} \

bhyveload -m ${ram} -c /dev/null -d ${disk} -e autoboot_delay=1 ${name}
#bhyveload -m ${ram} -c /dev/null -d ${installer} -e autoboot_delay=1 ${name}

bhyve \
        -c ${cpu} -m ${ram} \
        -A -H \
        -s 0,hostbridge \
        -s 2,ahci-hd,${disk} \
        -s 8,virtio-net,${tap} \
        -s 16,lpc \
        -l com1,${con} \
        ${name}