Skip to content

Instantly share code, notes, and snippets.

@mkhon
Last active Mar 31, 2021
Embed
What would you like to do?
This is the original zfs_revert-0.1.py script by Martin Vool <mardicas@gmail.com> for reverting ZFS transactions by destroying uberblocks, adapted for FreeBSD od (multiple spaces in output).
#!/usr/bin/python
# -*- coding: utf-8 -*-
#Script for reverting ZFS changes by destroying uberblocks
#Author: Martin Vool
#E-mail: mardicas@gmail.com
#Version: 0.1
#Date: 16 November 2009
import time
import subprocess
import sys
import os
#Default blocksize
bs=512
#default total blocks (sorry programming in estonian :-/)
suurus=None
if len(sys.argv) > 2:
for arg in sys.argv:
arg=arg.split('=')
if len(arg) == 1:
file=arg[0]
elif arg[0] == '-bs':
bs=int(arg[1])
elif arg[0] == '-tb':
suurus=int(arg[1])
else:
print 'Usage: zfs_revert.py [-bs=n default:n=512 blocksize] \\n [-tb=n total block size in blocks] [file/device] You have to set -tb'
exit(1)
print int(bs)
if suurus == None:
print 'Total block size in blocks is undefined'
exit(1)
#make solaris use gnu grep.
if os.uname()[0] == 'SunOS':
grep_cmd='ggrep'
else:
grep_cmd='grep'
#to format program output
def formatstd(inp):
inp=inp.split('\n')
ret=[]
for line in inp:
columns=line.split(' ')
nc=[]
for c in columns:
if c != '':
nc.append(c)
ret.append(nc)
return ret
#read blocks from beginning(64mb)
a_count=(256*bs)
#read blocks from end (64mb)
l_skip=suurus-(256*bs)
print 'Total of %s blocks'%suurus
print 'Reading from the beginning to %s blocks'%a_count
print 'Reading from %s blocks to the end'%l_skip
#get the uberblocks from the beginning and end
yberblocks_a=formatstd(subprocess.Popen('sync && dd bs=%s if=%s count=%s | od -A x -x | %s -A 2 "b10c *00ba" | %s -v "\-\-"'%(bs,file, a_count,grep_cmd,grep_cmd), shell=True, stdout=subprocess.PIPE).communicate()[0])
yberblocks_l=formatstd(subprocess.Popen('sync && dd bs=%s if=%s skip=%s | od -A x -x | %s -A 2 "b10c *00ba" | %s -v "\-\-"'%(bs,file, l_skip,grep_cmd,grep_cmd), shell=True, stdout=subprocess.PIPE).communicate()[0])
yberblocks=[]
for p in yberblocks_a:
if len(p) > 0:
#format the hex address to decmal so dd would eat it.
p[0]=(int(p[0], 16)/bs)
yberblocks.append(p)
for p in yberblocks_l:
if len(p) > 0:
#format the hex address to decmal so dd would eat it and add the skipped part.
p[0]=((int(p[0], 16)/bs)+int(l_skip)) #we have to add until the place we skipped so the adresses would mach.
yberblocks.append(p)
print '----'
#here will be kept the output that you will see later(TXG, timestamp and the adresses, should be 4, might be less)
koik={}
i=0
for p in yberblocks:
if len(p) > 0:
if i == 0:#the first output line
address=p[0]
elif i == 1:#second output line
#this is the output of od that is in hex and needs to be reversed
txg=int(p[4]+p[3]+p[2]+p[1], 16)
elif i == 2:#third output line
timestamp=int(p[4]+p[3]+p[2]+p[1], 16)
try:
aeg=time.strftime("%d %b %Y %H:%M:%S", time.localtime(timestamp))
except:
aeg='none'
if koik.has_key(txg):
koik[txg]['addresses'].append(address)
else:
koik[txg]={
'txg':txg,
'timestamp':timestamp,
'htime': aeg,
'addresses':[address]
}
if i == 2:
i=0
else:
i+=1
keys = koik.keys()
keys.sort()
while True:
keys = koik.keys()
keys.sort()
print 'TXG\tTIME\tTIMESTAMP\tBLOCK ADDRESSES'
for k in keys:
print '%s\t%s\t%s\t%s'%(k, koik[k]['htime'],koik[k]['timestamp'],koik[k]['addresses'])
try:
save_txg=int(input('What is the last TXG you wish to keep?\n'))
keys = koik.keys()
keys.sort()
for k in keys:
if k > save_txg:
for adress in koik[k]['addresses']:
#wrtie zeroes to the unwanted uberblocks
format=formatstd(subprocess.Popen('dd bs=%s if=/dev/zero of=%s seek=%s count=1 conv=notrunc'%(bs, file, adress), shell=True, stdout=subprocess.PIPE).communicate()[0])
#print('dd bs=%s if=/dev/zero of=%s seek=%s count=1 conv=notrunc'%(bs, file, adress))
del(koik[k])
#sync changes to disc!
#sync=formatstd(subprocess.Popen('sync', shell=True, stdout=subprocess.PIPE).communicate()[0])
except:
print ''
break
@mkhon

This comment has been minimized.

Copy link
Owner Author

@mkhon mkhon commented Oct 27, 2020

I used this script to fix kernel panic during "zpool import" (inspired by openzfs/zfs#6497):

panic: Solaris(panic): blkptr at 0xfffffe000ec45300 has invalid CHECKSUM 32"

zfs-invalid-checksum-panic

  • List of transactions can be obtained using "zdb -ul DEV | grep '\<txg =' | sort -k3"
  • Last good transaction can be found using "zdb -AAA -L -t TXG -bcdmue POOL"

Bad transactions will cause error messages printed on the console like this:

Traversing all blocks to verify metadata checksums ...

 148G completed ( 328MB/s) estimated time remaining: 0hr 11min 07sec        WARNING: blkptr at 0x82c6499c0 has invalid CHECKSUM 32
WARNING: blkptr at 0x82c6499c0 has invalid COMPRESS 0
Assertion failed: (zio->io_error != 0), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c, line 660.
Assertion failed: (zio->io_vd != NULL), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 3628.
Assertion failed: (bp->blk_pad[0] == 0), file /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 3880.
zdb_blkptr_cb: Got error 22 reading <47, 179570, 0, 166>  -- skipping
 157G completed ( 290MB/s) estimated time remaining: 0hr 12min 02sec

So I used binary search using "zdb -t TXG" to find a transaction that does not cause zdb to dump errors like above and then used the script to revert to this transaction.

The script was run on each of the storage devices in a mirrored pool:

python2.7 zfs_revert.py -tb=3890251757 /dev/ada0p3
python2.7 zfs_revert.py -tb=3890251757 /dev/ada1p3

-tb parameter is the storage device size in sectors (can be obtained by using "fdisk -l DEV" or "gpart show DEV")

Theoretically "zpool import -T TXG" could be used instead of this script but it runs for ages (it was running for 4 hours for me on 1.8T volume and I was not able to wait any longer) and then may fail with "one or more devices is currently unavailable" error: openzfs/zfs#1926

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment