Skip to content

Instantly share code, notes, and snippets.

@yoyosan
Last active January 7, 2018 17:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yoyosan/bd973b9da7bf2dbac29a39fb4ef86e5b to your computer and use it in GitHub Desktop.
Save yoyosan/bd973b9da7bf2dbac29a39fb4ef86e5b to your computer and use it in GitHub Desktop.
Recover after failed kernel upgrade on Ubuntu 16.04 and online.net box

Recovering after failed kernel upgrade(meltdownattack) on Ubuntu 16.04 on online.net box

What happened?

Recently, I've updated my Ubuntu 16.04 machine from online.net due to the meltdownattack.com.

But it didn't go so well 😢

The installation process yielded a couple of errors:

Running depmod.
update-initramfs: deferring update (hook will be called later)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-104-generic /boot/vmlinuz-4.4.0-104-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-104-generic /boot/vmlinuz-4.4.0-104-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-104-generic

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-4.4.0-104-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-104-generic.postinst line 1052.
dpkg: error processing package linux-image-4.4.0-104-generic (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-104-generic:
 linux-image-extra-4.4.0-104-generic depends on linux-image-4.4.0-104-generic; however:
  Package linux-image-4.4.0-104-generic is not configured yet.

dpkg: error processing package linux-image-extra-4.4.0-104-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-image-4.4.0-104-generic; however:
  Package linux-image-4.4.0-104-generic is not configured yet.
 linux-image-generic depends on linux-image-extra-4.4.0-104-generic; however:
  Package linux-image-extra-4.4.0-104-generic is not configured yet.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
Setting up linux-headers-4.4.0-104 (4.4.0-104.127) ...
Setting up linux-headers-4.4.0-104-generic (4.4.0-104.127) ...
Setting up linux-headers-generic (4.4.0.104.109) ...
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 4.4.0.104.109); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
Setting up linux-libc-dev:amd64 (4.4.0-104.127) ...
Setting up nodejs (8.9.4-1nodesource1) ...
Setting up python-pkg-resources (33.1.1-1+certbot~xenial+1) ...
Setting up python-setuptools (33.1.1-1+certbot~xenial+1) ...
Setting up python3-certifi (2017.4.17-2+ubuntu16.04.1+certbot+1) ...
Setting up python3-pkg-resources (33.1.1-1+certbot~xenial+1) ...
Setting up python3-chardet (3.0.4-1+ubuntu16.04.1+certbot+2) ...
Setting up python3-idna (2.5-1+ubuntu16.04.1+certbot+1) ...
Setting up python3-six (1.11.0-1+ubuntu16.04.1+certbot+1) ...
Setting up python3-urllib3 (1.21.1-1+ubuntu16.04.1+certbot+1) ...
Setting up python3-requests (2.18.1-1+ubuntu16.04.1+certbot+1) ...
Setting up unattended-upgrades (0.90ubuntu0.9) ...
Processing triggers for initramfs-tools (0.122ubuntu8.10) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-103-generic

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-4.4.0-103-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Errors were encountered while processing:
 linux-image-4.4.0-104-generic
 linux-image-extra-4.4.0-104-generic
 linux-image-generic
 linux-generic
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

This pretty much means that there's no space left on my /boot partition to install new kernels 😿

Sadly, I've rebooted thinking that there shouldn't be any problem. Wrong again. I could no longer connect through SSH to my machine so I was even more sad.

I've contacted the support department of online.net but they don't offer help with these matters.

Recovering

After a bit of investigation, I managed to find a proper solution. Here's how below:

  1. Switch your machine to rescue mode, from the console.
  2. Use the generated credentials to connect to your machine.
  3. Prepare to chroot into your machine partitions by running:
$ sudo mkdir /media
$ cd /media
$ sudo fdisk -l
Périphérique Amorçage      Start        Fin   Secteurs   Size Id Type
/dev/sda1    *              2048     409599     407552   199M 83 Linux
/dev/sda2                 409600 1002115071 1001705472 477,7G 83 Linux
/dev/sda3             1002115072 1004263423    2148352     1G 82 partition d'échange Linux / Solaris
/dev/sda4             1004263424 1953523711  949260288 452,7G 83 Linux
# /dev/sda1 is the /boot partition
# /dev/sda2 is the / partition
# /dev/sda4 is the /home partition
# First mount temporary api filesystems
$ sudo mount /dev/sda2 /media
$ sudo mount -t proc proc proc/
$ sudo mount --rbind /sys sys/
$ sudo mount --rbind /dev dev/
$ sudo mount --rbind /run run/
# Mount the rest of the partitions
$ sudo mount /dev/sda1 /media/boot
$ sudo mount /dev/sda4 /media/home
  1. chroot into your machine environment
$ sudo chroot /media /bin/bash
  1. Recover by uninstalling existing kernel packages:
# First check kernel packages to be removed
$ kernelver=$(uname -r | sed -r 's/-[a-z]+//')
$ dpkg -l linux-{image,headers}-"[0-9]*" | awk '/ii/{print $2}' | grep -ve $kernelver
# Remove the packages that take space
$ apt-get purge $(dpkg -l linux-{image,headers}-"[0-9]*" | awk '/ii/{print $2}' | grep -ve "$(uname -r | sed -r 's/-[a-z]+//')")
$ exit
  1. Important Unmount partitions
$ cd /
$ sudo umount --recursive /media
  1. Switch your machine to Normal mode now.

Profit 💰.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment