Skip to content

Instantly share code, notes, and snippets.

@mkhon
Last active November 7, 2023 11:12
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save mkhon/34d979c78077a20648456272d7f2cc15 to your computer and use it in GitHub Desktop.
Save mkhon/34d979c78077a20648456272d7f2cc15 to your computer and use it in GitHub Desktop.
This is the original zfs_revert-0.1.py script by Martin Vool <mardicas@gmail.com> for reverting ZFS transactions by destroying uberblocks, adapted for FreeBSD od (multiple spaces in output).
#!/usr/bin/python
# -*- coding: utf-8 -*-
#Script for reverting ZFS changes by destroying uberblocks
#Author: Martin Vool
#E-mail: mardicas@gmail.com
#Version: 0.1
#Date: 16 November 2009
import time
import subprocess
import sys
import os
#Default blocksize
bs=512
#default total blocks (sorry programming in estonian :-/)
suurus=None
if len(sys.argv) > 2:
for arg in sys.argv:
arg=arg.split('=')
if len(arg) == 1:
file=arg[0]
elif arg[0] == '-bs':
bs=int(arg[1])
elif arg[0] == '-tb':
suurus=int(arg[1])
else:
print 'Usage: zfs_revert.py [-bs=n default:n=512 blocksize] \\n [-tb=n total block size in blocks] [file/device] You have to set -tb'
exit(1)
print int(bs)
if suurus == None:
print 'Total block size in blocks is undefined'
exit(1)
#make solaris use gnu grep.
if os.uname()[0] == 'SunOS':
grep_cmd='ggrep'
else:
grep_cmd='grep'
#to format program output
def formatstd(inp):
inp=inp.split('\n')
ret=[]
for line in inp:
columns=line.split(' ')
nc=[]
for c in columns:
if c != '':
nc.append(c)
ret.append(nc)
return ret
#read blocks from beginning(64mb)
a_count=(256*bs)
#read blocks from end (64mb)
l_skip=suurus-(256*bs)
print 'Total of %s blocks'%suurus
print 'Reading from the beginning to %s blocks'%a_count
print 'Reading from %s blocks to the end'%l_skip
#get the uberblocks from the beginning and end
yberblocks_a=formatstd(subprocess.Popen('sync && dd bs=%s if=%s count=%s | od -A x -x | %s -A 2 "b10c *00ba" | %s -v "\-\-"'%(bs,file, a_count,grep_cmd,grep_cmd), shell=True, stdout=subprocess.PIPE).communicate()[0])
yberblocks_l=formatstd(subprocess.Popen('sync && dd bs=%s if=%s skip=%s | od -A x -x | %s -A 2 "b10c *00ba" | %s -v "\-\-"'%(bs,file, l_skip,grep_cmd,grep_cmd), shell=True, stdout=subprocess.PIPE).communicate()[0])
yberblocks=[]
for p in yberblocks_a:
if len(p) > 0:
#format the hex address to decmal so dd would eat it.
p[0]=(int(p[0], 16)/bs)
yberblocks.append(p)
for p in yberblocks_l:
if len(p) > 0:
#format the hex address to decmal so dd would eat it and add the skipped part.
p[0]=((int(p[0], 16)/bs)+int(l_skip)) #we have to add until the place we skipped so the adresses would mach.
yberblocks.append(p)
print '----'
#here will be kept the output that you will see later(TXG, timestamp and the adresses, should be 4, might be less)
koik={}
i=0
for p in yberblocks:
if len(p) > 0:
if i == 0:#the first output line
address=p[0]
elif i == 1:#second output line
#this is the output of od that is in hex and needs to be reversed
txg=int(p[4]+p[3]+p[2]+p[1], 16)
elif i == 2:#third output line
timestamp=int(p[4]+p[3]+p[2]+p[1], 16)
try:
aeg=time.strftime("%d %b %Y %H:%M:%S", time.localtime(timestamp))
except:
aeg='none'
if koik.has_key(txg):
koik[txg]['addresses'].append(address)
else:
koik[txg]={
'txg':txg,
'timestamp':timestamp,
'htime': aeg,
'addresses':[address]
}
if i == 2:
i=0
else:
i+=1
keys = koik.keys()
keys.sort()
while True:
keys = koik.keys()
keys.sort()
print 'TXG\tTIME\tTIMESTAMP\tBLOCK ADDRESSES'
for k in keys:
print '%s\t%s\t%s\t%s'%(k, koik[k]['htime'],koik[k]['timestamp'],koik[k]['addresses'])
try:
save_txg=int(input('What is the last TXG you wish to keep?\n'))
keys = koik.keys()
keys.sort()
for k in keys:
if k > save_txg:
for adress in koik[k]['addresses']:
#wrtie zeroes to the unwanted uberblocks
format=formatstd(subprocess.Popen('dd bs=%s if=/dev/zero of=%s seek=%s count=1 conv=notrunc'%(bs, file, adress), shell=True, stdout=subprocess.PIPE).communicate()[0])
#print('dd bs=%s if=/dev/zero of=%s seek=%s count=1 conv=notrunc'%(bs, file, adress))
del(koik[k])
#sync changes to disc!
#sync=formatstd(subprocess.Popen('sync', shell=True, stdout=subprocess.PIPE).communicate()[0])
except:
print ''
break
@mkhon
Copy link
Author

mkhon commented Oct 27, 2020

I used this script to fix kernel panic during "zpool import" (inspired by openzfs/zfs#6497):

panic: Solaris(panic): blkptr at 0xfffffe000ec45300 has invalid CHECKSUM 32"

zfs-invalid-checksum-panic

  • List of transactions can be obtained using "zdb -ul DEV | grep '\<txg =' | sort -k3"
  • Last good transaction can be found using "zdb -AAA -L -t TXG -bcdmue POOL"

Bad transactions will cause error messages printed on the console like this:

Traversing all blocks to verify metadata checksums ...

 148G completed ( 328MB/s) estimated time remaining: 0hr 11min 07sec        WARNING: blkptr at 0x82c6499c0 has invalid CHECKSUM 32
WARNING: blkptr at 0x82c6499c0 has invalid COMPRESS 0
Assertion failed: (zio->io_error != 0), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c, line 660.
Assertion failed: (zio->io_vd != NULL), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 3628.
Assertion failed: (bp->blk_pad[0] == 0), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 3880.
zdb_blkptr_cb: Got error 22 reading <47, 179570, 0, 166>  -- skipping
 157G completed ( 290MB/s) estimated time remaining: 0hr 12min 02sec

So I used binary search using "zdb -t TXG" to find a transaction that does not cause zdb to dump errors like above and then used the script to revert to this transaction.

The script was run on each of the storage devices in a mirrored pool:

python2.7 zfs_revert.py -tb=3890251757 /dev/ada0p3
python2.7 zfs_revert.py -tb=3890251757 /dev/ada1p3

-tb parameter is the storage device size in sectors (can be obtained by using "fdisk -l DEV" or "gpart show DEV")

Theoretically "zpool import -T TXG" could be used instead of this script but it runs for ages (it was running for 4 hours for me on 1.8T volume and I was not able to wait any longer) and then may fail with "one or more devices is currently unavailable" error: openzfs/zfs#1926

@robn
Copy link

robn commented Nov 7, 2023

Hi! A warning from your friendly neighbourhood OpenZFS developer and data recovery guy.

These days, you should not need this script. zpool import -T <txg> and its varints should work ok. If the import time for rewind has got you down, look into the spa_load_verify_data and spa_load_verify_metadata options. Of course, make good use of -o readonly=on and/or zdb before doing anything that might modify your pool.

If you are still absolutely determined to use this program, understand that it knows nothing about ZFS, it just does some math from on the -bs and -tb parameters you pass it and scribbles zeroes onto your disks. If you get those wrong, you're likely to make things worse. In particular, -bs defaults to 512, unsuitable for a 4K disk, and -tb is the device size in number of -bs-sized blocks.

But seriously, don't do this. Engage a data recovery organisation that specialises in ZFS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment