I added a 256GB mSATA SSD to my Lenovo x220 of which I planned to use a nice chunk as L2ARC for the SATA HDD. Even without a SATA disk present the SSD showed up as HDD2 in the BIOS.
Made a fresh installation of FreeBSD 10.2 amd64
with the memstick image and chose ZFS+Geli.
Booted, worked, everything just fine. I added a
SATA HDD which the BIOS listed as HDD0 just as
I expected from the previous SATA-only setup.
When I now tried to boot from the SSD I was
prompted for the GELI key of a different device
than before, ada1p3 instead of ada0p3.
But my passphrase wasn't accepted. Never.
Re-installed, added HDD, same result.
Reinstalled with HDD present, the removed the
HDD, same result (then prompted for ada0p3
instead of ada1p3, of course).
When I restored the configuration which was present during the installation process (either remove the HDD or add it back in) everything worked fine again.
Created a VM, added more disks:
floh@fuchi-cyber220:~:1305% iohyve info
Name Size RAM CPU OS Loader
fbsd102 - 2G 1 default bhyveload
fbsd102/disk0 2G 2G 1 default bhyveload
fbsd102/disk1 2G 2G 1 default bhyveload
fbsd102/disk2 4G 2G 1 default bhyveload
Install FreeBSD 10.2 (the one I used for my laptop):
floh@fuchi-cyber220:~:1305% sudo iohyve install fbsd102 FreeBSD-10.2-RELEASE-amd64-disc1.iso
Installing fbsd102...
floh@fuchi-cyber220:~:1307% sudo iohyve console fbsd102
Starting console on fbsd102...
~~. to escape console [uses cu(1) for console]
Connected
- Chose Auto (ZFS), added only ada2/disk2 to the rootpool.
┌────────────────ZFS Configuration───────────────────┐
│ Configure Options: │
│ ┌────────────────────────────────────────────────┐ │
│ │ >>> Install Proceed with Installation │ │
│ │ T Pool Type/Disks: stripe: 1 disk │ │
│ │ - Rescan Devices * │ │
│ │ - Disk Info * │ │
│ │ N Pool Name zroot │ │
│ │ 4 Force 4K Sectors? YES │ │
│ │ E Encrypt Disks? YES │ │
│ │ P Partition Scheme GPT │ │
│ │ S Swap Size 256m │ │
│ │ M Mirror Swap? NO │ │
│ │ W Encrypt Swap? YES │ │
│ └────────────────────────────────────────────────┘ │
├────────────────────────────────────────────────────┤
│ <Select> <Cancel> │
└────────────────────────────────────────────────────┘
-
proceed with installation.
-
remove disk1 from the VM:
floh@fuchi-cyber220:~:1313% sudo iohyve remove fbsd102 disk1 Are you sure you want to remove disk1 from fbsd102 [Y/N]? y floh@fuchi-cyber220:~:1314% iohyve info Name Size RAM CPU OS Loader fbsd102 - 2G 1 default bhyveload fbsd102/disk0 2G 2G 1 default bhyveload fbsd102/disk2 4G 2G 1 default bhyveload -
shuffle disks around:
floh@fuchi-cyber220:~:1340% sudo zfs rename zroot/iohyve/fbsd102/disk{0,1} floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{2,0} floh@fuchi-cyber220:~:1342% sudo zfs rename zroot/iohyve/fbsd102/disk{1,2} -
now the 4GB one is
disk0:floh@fuchi-cyber220:~:1343% iohyve info Name Size RAM CPU OS Loader fbsd102 - 2G 1 default bhyveload fbsd102/disk0 4G 2G 1 default bhyveload fbsd102/disk2 2G 2G 1 default bhyveload -
boot the VM and try to get the rootpool imported:
floh@fuchi-cyber220:~:1344% sudo iohyve start fbsd102
Starting fbsd102... (Takes 15 seconds for FreeBSD guests)
floh@fuchi-cyber220:~:1344%
floh@fuchi-cyber220:~:1345% sudo iohyve console fbsd102
Starting console on fbsd102...
~~. to escape console [uses cu(1) for console]
Connected
/
______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ ``` `
s` `.....---.......--.``` -/
+============Welcome to FreeBSD===========+ +o .--` /y:` +.
| | yo`:. :o `+-
| 1. Boot Multi User [Enter] | y/ -/` -o/
| 2. Boot [S]ingle User | .- ::/sy+:.
| 3. [Esc]ape to loader prompt | / `-- /
| 4. Reboot | `: :`
| | `: :`
| Options: | / /
| 5. [K]ernel: kernel (1 of 2) | .- -.
| 6. Configure Boot [O]ptions... | -- -.
| | `:` `:`
| | .-- `--.
| | .---.....----.
+=========================================+
/boot/kernel/kernel text=0xfc8de8 data=0x1283b0+0x207880 syms=[0x8+0x145350+0x8+0x15fe20]
/boot/kernel/zfs.ko size 0x2f9b00 at 0x199e000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x6048 at 0x1c98000
/boot/kernel/geom_eli.ko size 0x21568 at 0x1c9f000
loading required module 'crypto'
/boot/kernel/crypto.ko size 0x35318 at 0x1cc1000
/boot/encryption.key size=0x1000
/boot/kernel/aesni.ko size 0x5a30 at 0x1cf8000
/boot/zfs/zpool.cache size=0x8bc
Booting...
Copyright (c) 1992-2015 The FreeBSD Project.
[... lots kernel messages ...]
ada1: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada1: Serial Number BHYVE-D12E-A75C-27F5
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2048MB (4194304 512 byte sectors: 16H 63S/T 4161C)
ada1: Previously was known as ad6
random: unblocking device.
Timecounter "TSC-low" frequency 1345403547 Hz quality 1000
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 2.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. Tries left: 1.
Enter passphrase for ada0p4:
GEOM_ELI: Wrong key for ada0p4. No tries left.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 2.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. Tries left: 1.
Enter passphrase for diskid/DISK-BHYVE-F485-E5AF-7C59p4:
GEOM_ELI: Wrong key for diskid/DISK-BHYVE-F485-E5AF-7C59p4. No tries left.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 2.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. Tries left: 1.
Enter passphrase for gpt/zfs0:
GEOM_ELI: Wrong key for gpt/zfs0. No tries left.
Trying to mount root from zfs:zroot/ROOT/default []...
Mounting from zfs:zroot/ROOT/default failed with error 2.
Loader variables:
vfs.root.mountfrom=zfs:zroot/ROOT/default
Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with the specified (optional) option list.
eg. ufs:/dev/da0s1a
zfs:tank
cd9660:/dev/acd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/acd0 /)
? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abort manual input
mountroot>
panic: mountroot: unable to (re-)mount root.
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80984e30 at kdb_backtrace+0x60
#1 0xffffffff809489e6 at vpanic+0x126
#2 0xffffffff809488b3 at panic+0x43
#3 0xffffffff809e7f5f at vfs_mountroot+0x1eaf
#4 0xffffffff808f03b3 at start_init+0x53
#5 0xffffffff8091244a at fork_exit+0x9a
#6 0xffffffff80d30d2e at fork_trampoline+0xe
Uptime: 4m16s
And, no, srsly, I don't need more than 9 tries to type "foobar".
Changing the order of disks back should fix the problem. As there's no BIOS menu to choose the boot device in bhyve having no bootcode in (the original) disk0 makes this a tiny bit difficult with bhyve.
@allanjude: mentioned the issue some weeks ago on twitter.
Finally tried to reproduce the problem with bhyve and then opened a PR:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206521 ;)