Skip to content

Instantly share code, notes, and snippets.

@porjo
Created September 22, 2023 00:35
Show Gist options
  • Save porjo/117496483c51352a15e2737de95683ac to your computer and use it in GitHub Desktop.
Save porjo/117496483c51352a15e2737de95683ac to your computer and use it in GitHub Desktop.

My TrueNAS Scale server recently started throwing the following zfs module kernel panic on boot:

PANIC at space_map.c:405:space_map_load_callback()

What follows are the steps I took to retrieve the data from the corrupted zfs pool.

Getting setup

  • boot from a Ubuntu 22.04 Live USB disk. Ubuntu (unlike Fedora) supports zfs without needing to compile kernel modules.
  • install the zfsutils-linux package:
$ apt install zfsutils-linux
  • import the pool (Seagate4TB) as readonly (attempting to import read/write causes the kernel to panic!)
$ zpool import -o readonly=on -f Seagate4TB -R /mnt

Pool file systems are now available (readonly) under /mnt. In my case, I had a file system that I was using for NFS storage. Volumes have to be accessed a different way - read on to find out how!

Using a new pool

One complication I had is that the data I really needed was inside a zvol which had an XFS filesystem on it. Attempting to mount that directly from the readonly pool resulted in an XFS error due to it being read-only. The solution was to copy the zvol to a new pool.

The new pool needs to reside somewhere with sufficient free space to hold data exported from the readonly pool. I used a spare 500Gig 2.5" USB hard disk I had lying around. Instead of using the USB disk as a block device for the pool, I created a sparse file on the USB disk and used that for the pool's storage.

  • create a 100Gig sparse file
$ dd if=/dev/zero of=/external-usb/testpool bs=1 count=0 seek=100G
  • create the new pool on the sparse file
$ zpool create testpool /external-usb/testpool
  • export the zvol from the readonly pool to the newly created pool
$ zfs send Seagate4TB/Fedora-home | zfs receive testpool/Fedora-home
  • mount the zvol from the new pool to a temporary location
mount /dev/zvol/testpool/Fedora-home /mnt2
  • retrieve the needed files from /mnt2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment