00-info.md

ZFS: 2.1.5-1 Kernel: 5.18.19

Corruption happened on different bad hardware. Assume the disk is not faulty. It is now verified that the hardware that corrupted the pool is faulty. Recovery is done on another disk.

Read-only import works fine

# zpool import -f -o readonly=on big-storage

01-zdb.txt

# zdb -vvvvvv -b -e big-storage

Traversing all blocks to verify nothing leaked ...

loading concrete vdev 0, metaslab 6 of 232 ...error: zfs: removing nonexistent segment from range tree (offset=180fefe000 size=20000)
Aborted (core dumped)

02-import.txt

NOTE: command hangs, kernel panic shared after

~ # zpool import -f big-storage

 PANIC: zfs: adding existent segment to range tree (offset=180fefe000 size=20000)
 Showing stack for process 149045
 CPU: 2 PID: 149045 Comm: z_wr_iss Tainted: P           O      5.18.19 #1-NixOS
 Hardware name: ZOTAC XXXXXX/XXXXXX, BIOS 4.6.5 09/15/2015
 Call Trace:
  <TASK>
  dump_stack_lvl+0x45/0x5e
  vcmn_err.cold+0x50/0x68 [spl]
  ? zfs_btree_insert_core_impl.isra.0+0x76/0x90 [zfs]
  ? zfs_btree_insert_into_leaf+0x232/0x2a0 [zfs]
  ? zfs_btree_insert_into_leaf+0x232/0x2a0 [zfs]
  ? pn_free+0x30/0x30 [zfs]
  ? zfs_btree_find_parent_idx+0x72/0xd0 [zfs]
  zfs_panic_recover+0x6d/0x90 [zfs]
  range_tree_add_impl+0x303/0xe40 [zfs]
  ? zio_wait+0x260/0x290 [zfs]
  space_map_load_callback+0x55/0x90 [zfs]
  space_map_iterate+0x193/0x3d0 [zfs]
  ? spa_stats_destroy+0x190/0x190 [zfs]
  space_map_load_length+0x5e/0xe0 [zfs]
  metaslab_load+0x14d/0x8a0 [zfs]
  ? range_tree_add_impl+0x759/0xe40 [zfs]
  metaslab_activate+0x50/0x2b0 [zfs]
  ? preempt_count_add+0x70/0xa0
  metaslab_alloc_dva+0x351/0x1490 [zfs]
  metaslab_alloc+0xd3/0x280 [zfs]
  zio_dva_allocate+0xd3/0x900 [zfs]
  ? __kmalloc_node+0x17b/0x370
  ? preempt_count_add+0x70/0xa0
  ? _raw_spin_lock+0x13/0x40
  zio_execute+0x83/0x120 [zfs]
  taskq_thread+0x2cf/0x500 [spl]
  ? wake_up_q+0x90/0x90
  ? zio_gang_tree_free+0x70/0x70 [zfs]
  ? taskq_thread_spawn+0x60/0x60 [spl]
  kthread+0xe8/0x110
  ? kthread_complete_and_exit+0x20/0x20
  ret_from_fork+0x22/0x30
  </TASK>

03-import-with-options

NOTE: command hangs here too

~ # echo 1 > /sys/module/zfs/parameters/zfs_recover 

~ # echo 1 > /sys/module/zfs/parameters/zil_replay_disable 

~ # zpool import -f big-storage

 WARNING: zfs: adding existent segment to range tree (offset=180fefe000 size=20000)
 VERIFY3(rs_get_end(rs, rt) >= end) failed (103347642368 >= 103347757056)
 PANIC at range_tree.c:485:range_tree_remove_impl()
 Showing stack for process 3145
 CPU: 2 PID: 3145 Comm: z_wr_iss Tainted: P           O      5.18.19 #1-NixOS
 Hardware name: ZOTAC XXXXXX/XXXXXX, BIOS 4.6.5 09/15/2015
 Call Trace:
  <TASK>
  dump_stack_lvl+0x45/0x5e
  spl_panic+0xd1/0xe9 [spl]
  ? zfs_btree_insert_into_leaf+0x232/0x2a0 [zfs]
  ? __kmalloc_node+0x17b/0x370
  ? zfs_btree_insert_into_leaf+0x232/0x2a0 [zfs]
  ? zfs_btree_find_parent_idx+0x72/0xd0 [zfs]
  ? pn_free+0x30/0x30 [zfs]
  ? zfs_btree_find_parent_idx+0x72/0xd0 [zfs]
  ? zfs_btree_find+0x175/0x300 [zfs]
  range_tree_remove_impl+0xc6c/0xef0 [zfs]
  ? zio_wait+0x260/0x290 [zfs]
  space_map_load_callback+0x22/0x90 [zfs]
  space_map_iterate+0x193/0x3d0 [zfs]
  ? spa_stats_destroy+0x190/0x190 [zfs]
  space_map_load_length+0x5e/0xe0 [zfs]
  metaslab_load+0x14d/0x8a0 [zfs]
  ? range_tree_add_impl+0x759/0xe40 [zfs]
  metaslab_activate+0x50/0x2b0 [zfs]
  ? preempt_count_add+0x70/0xa0
  metaslab_alloc_dva+0x351/0x1490 [zfs]
  metaslab_alloc+0xd3/0x280 [zfs]
  zio_dva_allocate+0xd3/0x900 [zfs]
  ? __kmalloc_node+0x17b/0x370
  ? preempt_count_add+0x70/0xa0
  ? _raw_spin_lock+0x13/0x40
  zio_execute+0x83/0x120 [zfs]
  taskq_thread+0x2cf/0x500 [spl]
  ? wake_up_q+0x90/0x90
  ? zio_gang_tree_free+0x70/0x70 [zfs]
  ? taskq_thread_spawn+0x60/0x60 [spl]
  kthread+0xe8/0x110
  ? kthread_complete_and_exit+0x20/0x20
  ret_from_fork+0x22/0x30