| title | author | weight | draft |
|---|---|---|---|
Grub issues that you need to be aware of |
Ed Vielmetti |
1 |
true |
Recent changes and updates to the GRUB boot loader on the Ubuntu operating system have lead to a temporary condition where some Ubuntu systems will fail to reboot correctly. While we at Equinix Metal are working on a permanent fix, here's what can go wrong and how you can work around it.
If an affected system is rebooted, it might not come back online. Instead, the serial over SSH or SOS console will show the system at the GRUB prompt.
To recover from this condition, log in to the SOS console, which will connect you to GRUB. Then issue the following command:
grub> configfile ($root)/EFI/GRUB/grub.cfg
The device will load the correct boot sequence and return to service.
Before a system goes through its next planned boot cycle, you can update Ubuntu to correct the boot sequence so that it successfully reboots.
First, update GRUB to the latest version:
sudo apt-get update
sudo apt-get install grub2-common
Then, tell GRUB to follow the Ubuntu boot sequence.
% sudo grub-install --bootloader-id=ubuntu
Installing for arm64-efi platform.
Installation finished. No error reported.
After following these instructions, the system should reboot normally.
GRUB has been recently updated to address a series of security issues. You can read about improvements in GRUB security to understand the details.
One of the changes to GRUB as a result is the use of cryptographically signed EFI binaries. The old GRUB update process no longer works correctly because the cryptographic signatures prevent the necessary changes. By following the above instructions, you properly handle the bootloader update case.
Equinix Metal engineers are rolling out a fix in a future Ubuntu operating system upgrade. Brand new installations will not need any of these instructions. People with existing systems may still need to undertake preventative measures. Updated details will be shared when the process is complete.
I ran into this problem on Ubuntu 22.04 servers built in November 2022, so the "Brand new installations will not need any of these instructions" may not be entirely accurate.
One of my servers booted into a
grub>prompt after auto rebooting after unattended upgrades. I've since run the mitigation steps and so far things appear to be OK, but I haven't yet rebooted the problematic server.