bug detail: mdadm may broke bbl if bitmap is too large

cleanup.sh

#!/bin/sh -ux

. ./variables.sh

for version in xenial zesty artful
do
    MDADM=$ARCHIVES/$version/contents/sbin/mdadm
    sudo $MDADM -S /dev/md/$version
    sudo $MDADM -r /dev/md/$version
    sudo $MDADM -S /dev/md/${version}_nobb
    sudo $MDADM -r /dev/md/${version}_nobb
done

sudo dmsetup remove /dev/mapper/$DMNAME

for i in 0 1 2 3
do
    sudo losetup -d /dev/loop$i
done

mdadm_bug_bbl_disruption.md

Attached scripts will verify:

• mdadm 3.4-4ubuntu0.1 (for zesty) puts a bbl on a bitmap and causes disruption, and
• mdadm 4.0-2 (for artful) adjusts position of a bbl according to bitmap size.

experiment scripts

I've attached some scripts to reproduce the bug. These scripts are tested with Ubuntu xenial.

1. prepare.sh : prepare for experiments
   • makes directories
   • downloads mdadm deb packages
2. procedure.sh : run an experiment
   • creates loopback devices which are used as MD components
   • creates MD devices
   • examines bbl
3. cleanup.sh : cleanup an experiment
   • deletes MD devices
   • deletes loopback devices
4. variables.sh : definition of common variables

detailed description of the bug

procedure.sh creates a bitmap with the size calculated as the bitmap can overlap with a bbl. It is expected that a bitmap overlaps with a bbl by using mdadm 3.4-4ubuntu0.1 (for zesty), doesn't overlap by using mdadm 4.0-2 (for artful).

./procedure.sh zesty creates block devices which will be used as MD components, then assembles MD devices.

After assembling, it shows the result of mdadm --examine and mdadm --examine-badblocks.

+ sudo /home/kota_uchida/mdadm/archives/zesty/contents/sbin/mdadm -E /dev/loop0
/dev/loop0:
...
   Super Offset : 8 sectors
...
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
...

+ sudo /home/kota_uchida/mdadm/archives/zesty/contents/sbin/mdadm --examine-badblocks /dev/mapper/disk_with_badblock
Bad-blocks on /dev/mapper/disk_with_badblock:
               17384 for 8 sectors

The result shows:

• The bbl is places at 80 sectors from the beginning of /dev/loop0.
  • Offset of the bbl is 72 sectors from the super block.
  • Offset of the super block is 8 sectors from the beginning of the device.
• Only one entry is in the bbl just after the MD device was assembled and MD detects a bad sector correctly.
  • One bad sector has been created at 17384 sector. See procedure.sh.

After 20 seconds, the bbl will be in inconsistent status.

+ sleep 20
+ tail -n 5
+ sudo /home/kota_uchida/mdadm/archives/zesty/contents/sbin/mdadm --examine-badblocks /dev/mapper/disk_with_badblock
                   0 for 0 sectors
                   0 for 0 sectors
                   0 for 0 sectors
                   0 for 0 sectors
                   0 for 0 sectors

You can test the bug with mdadm 4.0-2 (for artful) by running ./cleanup.sh and then ./procedure.sh artful.

With mdadm 4.0-2 (for artful), the bug shall not occur because a bbl will be created at a position outside a bitmap.

prepare.sh

#!/bin/sh -uex

. ./variables.sh

rm -rf $DISKS
rm -rf $ARCHIVES

mkdir $DISKS
mkdir $ARCHIVES
mkdir $ARCHIVES/xenial
mkdir $ARCHIVES/zesty
mkdir $ARCHIVES/artful

wget -P $ARCHIVES/xenial http://launchpadlibrarian.net/344961441/mdadm_3.3-2ubuntu7.6_amd64.deb
wget -P $ARCHIVES/zesty http://launchpadlibrarian.net/341565971/mdadm_3.4-4ubuntu0.1_amd64.deb
wget -P $ARCHIVES/artful http://launchpadlibrarian.net/340479220/mdadm_4.0-2_amd64.deb

(cd $ARCHIVES/xenial; dpkg-deb -x mdadm_*.deb contents)
(cd $ARCHIVES/zesty; dpkg-deb -x mdadm_*.deb contents)
(cd $ARCHIVES/artful; dpkg-deb -x mdadm_*.deb contents)

procedure.sh

#!/bin/sh -uex

. ./variables.sh

GIB=10
BMCHUNK=32

if [ $# -ne 1 ]
then
    echo "Usage: $0 xenial|zesty|artful"
    exit 1
fi

VERSION=$1

LO=0
for md in $VERSION ${VERSION}_nobb
do
    mkdir -p $DISKS/$md
    dd of=$DISKS/$md/sda if=/dev/zero bs=1073741824 count=$GIB
    dd of=$DISKS/$md/sdb if=/dev/zero bs=1073741824 count=$GIB
    #dd of=$DISKS/$md/sda if=/dev/zero bs=1048576 count=16 conv=notrunc
    #dd of=$DISKS/$md/sdb if=/dev/zero bs=1048576 count=16 conv=notrunc

    sudo losetup /dev/loop$LO $DISKS/$md/sda
    LO=$(($LO + 1))
    sudo losetup /dev/loop$LO $DISKS/$md/sdb
    LO=$(($LO + 1))
done

SECTORS=$(($GIB * 1024 * 1024 * 2))
BAD_SECTOR=17384

cat > $DMNAME.def << EOF
 0 $BAD_SECTOR  linear $UNDERDEV 0
 $BAD_SECTOR 1  error
 $(($BAD_SECTOR + 1)) $(($SECTORS - $BAD_SECTOR - 1))  linear $UNDERDEV $(($BAD_SECTOR + 1))
EOF

cat $DMNAME.def
sudo dmsetup create $DMNAME $DMNAME.def

MDADM=$ARCHIVES/$VERSION/contents/sbin/mdadm
MDADM_OPTS="--auto=md --symlink=no -n 2 -l 1 -e 1.2 -b internal --bitmap-chunk=$BMCHUNK"

sudo $MDADM -C /dev/md/$VERSION $MDADM_OPTS /dev/mapper/$DMNAME /dev/loop1
sudo $MDADM -C /dev/md/${VERSION}_nobb $MDADM_OPTS /dev/loop2 /dev/loop3
sudo $MDADM -E /dev/loop0

sudo $MDADM --wait /dev/md/$VERSION || true
sudo $MDADM --wait /dev/md/${VERSION}_nobb || true

sudo $MDADM --examine-badblocks /dev/mapper/disk_with_badblock | tail -n 5
sudo $MDADM --examine-badblocks /dev/loop2 | tail -n 5
sleep 20
sudo $MDADM --examine-badblocks /dev/mapper/disk_with_badblock | tail -n 5
sudo $MDADM --examine-badblocks /dev/loop2 | tail -n 5

variables.sh

DISKS=$PWD/disks
ARCHIVES=$PWD/archives

DMNAME=disk_with_badblock
UNDERDEV=/dev/loop0