biggers/linux_rescue_notes.rst

## linux_rescue_notes.rst

      
    Raw
  

              linux_rescue_notes.rst
            
          
    Rescuing a Linux System


Author
Mark Biggers <biggers@utsl.com>

Description
How to rescue a Linux system (OpenSUSE 12.x)

Ref
System Rescue CD docs: <http://www.sysresccd.org/Online-Manual-EN>

Ref
chroot, explained: <http://en.wikipedia.org/wiki/Chroot>

Ref
Change Root (prepare for rescue): <https://wiki.archlinux.org/index.php/Change_Root>

Ref
OpenSUSE wiki, "Recover root password": <https://en.opensuse.org/SDB:Recover_root_password>

Ref
Ubuntu LiveCdRecovery: <https://help.ubuntu.com/community/LiveCdRecovery>

Ref
xyzzy: <>

Revision
1.0

To View
restview README.rst

Metainfo
restview, Restructured Text Viewer: <https://pypi.python.org/pypi/nrestview>

Metainfo
Introductory ReST docs

Organization
UTSL.com, consulting DevOps & Python

Date
8 Nov 2013


Table of Contents


Getting Started

Review the documentation

Linux Recovery and Rescue documentation is available, as seen above. However, it tends to focus on a particular Rescue task, such as recovering a lost root password.
It's a good idea to read several Blogs or Linux articles that have solved various Rescue issues.
Usually, the Linux system Rescue process is not Linux distro-specific, except perhaps in the usage of a particular distro's Rescue boot, from a Live-CD, for example.
Create a Rescue "CD"

System Rescue CD can be burned to CD media, or easily copied via dd to USB thumb drive. A USB thumb drive will boot and operate much faster than a CD, and will also work on Ultrabooks (desktop) and servers, without a DVD drive.
How to copy to a USB drive, for System Rescue CD: :
mkdir -p /tmp/cdrom
sudo mount -o loop,exec /path/to/systemrescuecd-x86-x.y.z.iso /tmp/cdrom

cd /tmp/cdrom
sudo bash ./usb_inst.s
Read the full details, at:

http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick

A fallback is a Linux distro's Rescue mode. However, that particular distro may not have all the toolset included with the System Rescue CD.
Prepare for the Rescue

Mount filesystems to a chroot

This is modified, from the Arch Linux rescue-doc above. The plan is to create a Linux chroot, mounted under /mnt. When this is complete, we "change root" into this mount-directory, and an operate as if this is the current, booted filesystem - including running commands for Rescue purposes.
Activate any LVM volume-groups and logical volumes, first. Next, mount the root and boot filesystems (if boot is separate). Then mount all other necessary filesystems - usually var and usr, if these are separate filesystems from root. :
# activate all LVM components
vgchange -ay

# list all LVM volume-groups
vgs

# list all logical-volumes; determine "role" (root, usr, ... filesystems)
lvs

mount /dev/sysvg00/root  /mnt       # mount the root filesystem, first!
mount /dev/sda1          /mnt/boot  # ... presumes /boot is Linux partition !!

mount /dev/sysvg00/varlv   /mnt/var
mount /dev/sysvg00/usrlv   /mnt/usr
Now, mount the Linux "special" filesystems, in the chroot. :
cd /mnt

mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/

mount -t devpts pts dev/pts/

# if a "gnome-terminal cannot get permission" occurs, use this 'devpts' mount
mount -t devpts -o rw,nosuid,noexec,gid=5,mode=620,ptmxmode=000 devpts dev/pts
Finally, "change root" to the filesystem in the chroot, to begin running Rescue commands. :
sudo chroot /mnt   # opens a new shell, within the chroot
Unmount chroot filesystems

When the Rescue attempt is complete, all the chroot filesystems must be unmounted.
Basically, it is necessary to (just) reverse the order of the above mounts, with umount. When that is done, reboot the system and remove the Rescue CD / USB drive! :
exit           # "exit" from the chroot

cd /mnt
for fs in proc sysfs bind dev/pts dev; do  umount $fs; done
And so, un-mounts are also done for the "real" filesystems. :
cd /mnt
for fs in usr var boot; do  umount /mnt/$fs; done

cd /
umount /mnt/root
Rescue Scenario: broken video

The Rescue Plan

In this case, a Lenovo Thinkpad W530s is running OpenSUSE 12.3. A "set" of NVIDIA video drivers were installed, in an attempt to get "external LCD monitor support" working. Unfortunately, the video driver packages did not work, at all.
It is necessary to:


remove the NVIDIA driver packages, with zypper (OpenSUSE package manager)
rebuild the initramfs, with mkinitrd, so the old drivers are not present at (re)boot time
exit the chroot (shell), umount all chroot filesystems, then reboot


Doing the Rescue

First, remove the NVIDIA driver packages. It is helpful, if it's known what packages were installed; otherwise an "educated guess"" is needed. :
chroot /mnt

rpm -qa | egrep -i 'nvidia|dkms' | less   # an educated guess; a bit risky

zypper help remove

zypper remove $(rpm -qa | egrep -i 'nvidia|dkms')  # ASSUMES all these need removal!
Next, rebuild the "initial ram filesystem", initramfs for the Linux kernel. :
mkinitrd
Finally, exit the chroot, unmount the chroot's filesystems, and reboot.