I have two image files that I've created with makefs and mkimg. When I attach them both to a bhyve instance and boot it for the first time, it fails to boot.
When I boot with only the first image, it boots correctly. I can then shut it down and boot a second time attaching the second image, and it boots but does not recognize the zpool in the second image.
| ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 | |
| ada0: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device | |
| ada0: Serial Number BHYVE-94B0-8A9D-6C45 | |
| ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) | |
| ada0: Command Queueing enabled | |
| ada0: 21504MB (44040192 512 byte sectors) | |
| ada1 at ahcich32 bus 0 scbus1 target 0 lun 0 | |
| ada1: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device | |
| ada1: Serial Number BHYVE-83FA-5BCA-3020 | |
| ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) | |
| ada1: Command Queueing enabled | |
| ada1: 2048MB (4194304 512 byte sectors) | |
| Trying to mount root from zfs:zroot/ROOT/default []... | |
| Mounting from zfs:zroot/ROOT/default failed with error 22; retrying for 3 more seconds | |
| random: unblocking device. | |
| Mounting from zfs:zroot/ROOT/default failed with error 22. | |
| Loader variables: | |
| vfs.root.mountfrom=zfs:zroot/ROOT/default | |
| Manual root filesystem specification: | |
| <fstype>:<device> [options] | |
| Mount <device> using filesystem <fstype> | |
| and with the specified (optional) option list. | |
| eg. ufs:/dev/da0s1a | |
| zfs:zroot/ROOT/default | |
| cd9660:/dev/cd0 ro | |
| (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /) | |
| ? List valid disk boot devices | |
| . Yield 1 second (for background tasks) | |
| <empty line> Abort manual input | |
| mountroot> |
| root@builder:~ # gpart show | |
| => 34 44040125 ada0 GPT (21G) | |
| 34 102400 1 efi (50M) | |
| 102434 41943040 2 freebsd-zfs (20G) | |
| 42045474 1994685 - free - (974M) | |
| => 34 4194237 ada1 GPT (2.0G) | |
| 34 2097152 1 freebsd-zfs (1.0G) | |
| 2097186 2097085 - free - (1.0G) | |
| root@builder:~ # gpart list | |
| Geom name: ada0 | |
| modified: false | |
| state: OK | |
| fwheads: 16 | |
| fwsectors: 63 | |
| last: 44040158 | |
| first: 34 | |
| entries: 128 | |
| scheme: GPT | |
| Providers: | |
| 1. Name: ada0p1 | |
| Mediasize: 52428800 (50M) | |
| Sectorsize: 512 | |
| Stripesize: 131072 | |
| Stripeoffset: 17408 | |
| Mode: r0w0e0 | |
| efimedia: HD(1,GPT,c4478374-a577-11ef-b252-04421aacf71e,0x22,0x19000) | |
| rawuuid: c4478374-a577-11ef-b252-04421aacf71e | |
| rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b | |
| label: esp | |
| length: 52428800 | |
| offset: 17408 | |
| type: efi | |
| index: 1 | |
| end: 102433 | |
| start: 34 | |
| 2. Name: ada0p2 | |
| Mediasize: 21474836480 (20G) | |
| Sectorsize: 512 | |
| Stripesize: 131072 | |
| Stripeoffset: 17408 | |
| Mode: r1w1e1 | |
| efimedia: HD(2,GPT,c4478376-a577-11ef-b252-04421aacf71e,0x19022,0x2800000) | |
| rawuuid: c4478376-a577-11ef-b252-04421aacf71e | |
| rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b | |
| label: (null) | |
| length: 21474836480 | |
| offset: 52446208 | |
| type: freebsd-zfs | |
| index: 2 | |
| end: 42045473 | |
| start: 102434 | |
| Consumers: | |
| 1. Name: ada0 | |
| Mediasize: 22548578304 (21G) | |
| Sectorsize: 512 | |
| Stripesize: 131072 | |
| Stripeoffset: 0 | |
| Mode: r1w1e2 | |
| Geom name: ada1 | |
| modified: false | |
| state: OK | |
| fwheads: 16 | |
| fwsectors: 63 | |
| last: 4194270 | |
| first: 34 | |
| entries: 128 | |
| scheme: GPT | |
| Providers: | |
| 1. Name: ada1p1 | |
| Mediasize: 1073741824 (1.0G) | |
| Sectorsize: 512 | |
| Stripesize: 131072 | |
| Stripeoffset: 17408 | |
| Mode: r0w0e0 | |
| efimedia: HD(1,GPT,3906d994-a580-11ef-b252-04421aacf71e,0x22,0x200000) | |
| rawuuid: 3906d994-a580-11ef-b252-04421aacf71e | |
| rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b | |
| label: (null) | |
| length: 1073741824 | |
| offset: 17408 | |
| type: freebsd-zfs | |
| index: 1 | |
| end: 2097185 | |
| start: 34 | |
| Consumers: | |
| 1. Name: ada1 | |
| Mediasize: 2147483648 (2.0G) | |
| Sectorsize: 512 | |
| Stripesize: 131072 | |
| Stripeoffset: 0 | |
| Mode: r0w0e0 |
| # on the VM... which can't see zdata to import | |
| zdata: | |
| version: 5000 | |
| name: 'zdata' | |
| state: 0 | |
| txg: 4 | |
| pool_guid: 14298710786338440247 | |
| errata: 0 | |
| hostid: 4153604867 | |
| hostname: 'builder' | |
| com.delphix:has_per_vdev_zaps | |
| vdev_children: 1 | |
| vdev_tree: | |
| type: 'root' | |
| id: 0 | |
| guid: 14298710786338440247 | |
| create_txg: 4 | |
| com.klarasystems:vdev_zap_root: 129 | |
| children[0]: | |
| type: 'disk' | |
| id: 0 | |
| guid: 3777907824689759127 | |
| path: '/dev/ada1' | |
| whole_disk: 1 | |
| metaslab_array: 256 | |
| metaslab_shift: 30 | |
| ashift: 9 | |
| asize: 128844300288 | |
| is_log: 0 | |
| create_txg: 4 | |
| com.delphix:vdev_zap_leaf: 130 | |
| com.delphix:vdev_zap_top: 131 | |
| features_for_read: | |
| com.delphix:hole_birth | |
| com.delphix:embedded_data | |
| com.klarasystems:vdev_zaps_v2 | |
| zroot: | |
| version: 5000 | |
| name: 'zroot' | |
| state: 0 | |
| txg: 53 | |
| pool_guid: 4016146626377348012 | |
| errata: 0 | |
| hostname: 'builder' | |
| com.delphix:has_per_vdev_zaps | |
| vdev_children: 1 | |
| vdev_tree: | |
| type: 'root' | |
| id: 0 | |
| guid: 4016146626377348012 | |
| create_txg: 4 | |
| children[0]: | |
| type: 'disk' | |
| id: 0 | |
| guid: 100716240520803340 | |
| path: '/dev/ada0p2' | |
| whole_disk: 1 | |
| metaslab_array: 2 | |
| metaslab_shift: 29 | |
| ashift: 12 | |
| asize: 32154845184 | |
| is_log: 0 | |
| create_txg: 4 | |
| expansion_time: 1731917778 | |
| com.delphix:vdev_zap_leaf: 155 | |
| com.delphix:vdev_zap_top: 156 | |
| features_for_read: | |
| ZFS_DBGMSG(zdb) START: | |
| metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic | |
| metaslab.c:1682:spa_set_allocator(): spa allocator: dynamic | |
| ZFS_DBGMSG(zdb) END |