Buffered write to LTO tape on linux

write-to-tape.md

Writing to LTO tape with a text index and a memory buffer

This script is for writing to LTO tape on Linux with tar and producing a plain text index file. It uses a large memory buffer to keep the destination tape drive fed when the source drive can't consistently match the tape's write speed (eg. when there are a mix of small and large files).

This was written for a small, manually managed tape collection. It writes tape index information and tar listings as text to a file. This file can be stored separately, searched easily, and appended to the end of the tape if you want your tapes to be self-describing:

...
File number=12, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-25 12:10 vhs_tape_006/
-rwxrwxr-x stecman/stecman 57307360936 2020-05-25 11:50 vhs_tape_006/vhs_tape_006_capture_20200525_092527.mkv
File number=13, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-26 13:02 vhs_tape_007/
-rwxrwxr-x stecman/stecman 141489414757 2020-05-26 18:09 vhs_tape_007/vhs_tape_007_capture_20200526_130231.mkv
File number=14, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-26 18:16 vhs_tape_008/
-rwxrwxr-x stecman/stecman 139799858645 2020-05-26 23:44 vhs_tape_008/vhs_tape_008_capture_20200526_181559.mkv
...

At the start of each file, drive status and tape index will be printed:

SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x5a (LTO-6).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN
    IBM       ULTRIUM-HH6       H991
Tape capacity page  (LTO-5 and LTO-6 specific) [0x31]
  Main partition remaining capacity (in MiB): 2384185
  Alternate partition remaining capacity (in MiB): 0
  Main partition maximum capacity (in MiB): 2384185
  Alternate partition maximum capacity (in MiB): 0

While running, mbuffer provides a readout of the buffer input and output speeds:

in @  160 MiB/s, out @  160 MiB/s,  132 GiB total, buffer 100% full

Usage

Change the OWNER, TMP_LISTING and NEW_LISTING variables as needed before starting. Note that it's up to you to position the tape at the first file number you want written. The script does not rewind or go to end of tape automatically.

sudo ./write-to-tape.sh <FILE, ...>

The script will write each argument to a new tar file on the tape, each with a different file number. It's best to point it at logical chunks in directories rather than individual files. The tape position is not altered on start: writing will occur at the currently selected file number (mt -f /dev/nst0 status to check).

When the script ends, the tape position is left at the next available file number. The script can be called multiple times to keep adding items to a tape.

Reading back off the tape

# Position the tape at the desired file number
sudo mt -f /dev/nst0 asf 12

# Extract from the archive
# (Why this uses dd: https://unix.stackexchange.com/a/366217)
sudo dd if=/dev/nst0 bs=1M | tar xvf -

Other notes

• It's possible to do a partial tar extraction from a position on the tape, but it requires reading from at least the beginning of the archive up to the files you want to extract: the tape can't jump to a file in the tar archive unless you know its byte offset. Due to this it's best to write archives for logical chunks that you are likely to want to restore as a whole (and have the drive space required to do so).

• /dev/nst0 is used instead of /dev/st0 in all commands as it is the "no rewind" variant of the device. Using /dev/st0 causes an automatic rewind after every operation, which isn't helpful here.

write-to-tape.sh

#!/bin/bash

if [[ $(id -u) -ne 0 ]]; then
    echo "This script must be run as root"
    exit 1
fi

OWNER=yourusername
TMP_LISTING=/home/$OWNER/tapes/.new.txt
NEW_LISTING=/home/$OWNER/tapes/new.txt

for arg; do
	echo
	echo '---- Start:' $(date);
	mt -f /dev/nst0 status;

	mt -f /dev/nst0 status | grep File >> $NEW_LISTING;

	sg_logs /dev/nst0 -p tc_;
	du "$arg" && tar --index-file=$TMP_LISTING --blocking-factor=512 -cvvf - "$arg" | \
		mbuffer -m 10G -L -P 80 | \
		dd of=/dev/nst0 bs=256K iflag=fullblock;

	cat $TMP_LISTING >> $NEW_LISTING && rm $TMP_LISTING

	echo '---- End:' $(date);
done;

echo
echo '===== End all =====' $(date);
mt -f /dev/nst0 status;
sg_logs /dev/nst0 -p tc_;

chown $OWNER:$OWNER $NEW_LISTING

echo
echo "File index created: $NEW_LISTING"
echo