Skip to content

Instantly share code, notes, and snippets.

@gordonhart
Last active March 4, 2019 18:57
Show Gist options
  • Save gordonhart/cde25334fdc4f26cac90d116e753f4a0 to your computer and use it in GitHub Desktop.
Save gordonhart/cde25334fdc4f26cac90d116e753f4a0 to your computer and use it in GitHub Desktop.
Broken system recovery: chroot

Using a recovery medium + chroot to fix a broken Linux system

System-breaking changes happen when mucking around as root on a linux system.

Fixing a system that won't boot into a serviceable state is often a simple operation. Provided you have an idea of the operation you performed before rebooting that broke the installation, all it takes is an alternative bootable drive, say a USB with an OS installer. Many default installation .isos don't come with a full-enough featured enough shell to perform fixes but many others do ("live" installers). This guide details how to use a distro's installer shell to fix a broken installation of an arbitrary distribution, for example using a NixOS installation image to fix a broken Debian installation.

Show block devices to find the identifiers (e.g. /dev/sda1) of the broken system's partitions:

# lsblk

Knowledge of the partition scheme on the broken system is required although sometimes you can get away with an incomplete mounting of the broken system's drives.

Mount the broken OS drives (exact steps may vary):

# mount /dev/sda2 /mnt
# mount /dev/sda1 /mnt/boot/efi
# mount /dev/md0 /mnt/var

Note that some additional steps are required prior to mounting if your system's partitions are encrypted.

Certain directories found in / require special treatment as they are kernel-generated at runtime rather than living on storage media. Mount the necessary special file systems to have a full environment in the chroot:

# for d in "proc" "proc/sys" "sys" "dev"; do mount --bind /$d /mnt/$d; done

Change root directory (chroot) into /mnt to operate as root as if you were logged into the broken system:

# chroot /mnt /bin/bash

Note how we specify the shell executable to run in the chroot — in the case of a NixOS installer chrooting into a Debian system, the default bash executable lies at /run/current-system/sw/bin/bash on NixOS which is not available on Debian. Specify /bin/bash (relative to the root pointed to in the chroot command, here /mnt) to use the standard shell included on the broken Debian install.

Depending on the extent of the mismatch between the recovery system and broken system it may be necessary to add some standard paths to PATH:

# export PATH="$PATH:/bin:/usr/bin:/sbin:/usr/sbin"

Now you can make any changes necessary to recover the broken system, including apt (or other package manager operations), update-grub, etc.

Once changes are complete:

# exit
# reboot

And enjoy using the repaired system again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment