Skip to content

Instantly share code, notes, and snippets.

Last active March 19, 2024 14:08
Show Gist options
  • Save stecman/0150d84b521a2936ceb976e9f17865b7 to your computer and use it in GitHub Desktop.
Save stecman/0150d84b521a2936ceb976e9f17865b7 to your computer and use it in GitHub Desktop.
Buffered write to LTO tape on linux

Writing to LTO tape with a text index and a memory buffer

This script is for writing to LTO tape on Linux with tar and producing a plain text index file. It uses a large memory buffer to keep the destination tape drive fed when the source drive can't consistently match the tape's write speed (eg. when there are a mix of small and large files).

This was written for a small, manually managed tape collection. It writes tape index information and tar listings as text to a file. This file can be stored separately, searched easily, and appended to the end of the tape if you want your tapes to be self-describing:

File number=12, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-25 12:10 vhs_tape_006/
-rwxrwxr-x stecman/stecman 57307360936 2020-05-25 11:50 vhs_tape_006/vhs_tape_006_capture_20200525_092527.mkv
File number=13, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-26 13:02 vhs_tape_007/
-rwxrwxr-x stecman/stecman 141489414757 2020-05-26 18:09 vhs_tape_007/vhs_tape_007_capture_20200526_130231.mkv
File number=14, block number=0, partition=0.
drwxrwxr-x stecman/stecman   0 2020-05-26 18:16 vhs_tape_008/
-rwxrwxr-x stecman/stecman 139799858645 2020-05-26 23:44 vhs_tape_008/vhs_tape_008_capture_20200526_181559.mkv

At the start of each file, drive status and tape index will be printed:

SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x5a (LTO-6).
Soft error count since last status=0
General status bits on (41010000):
    IBM       ULTRIUM-HH6       H991
Tape capacity page  (LTO-5 and LTO-6 specific) [0x31]
  Main partition remaining capacity (in MiB): 2384185
  Alternate partition remaining capacity (in MiB): 0
  Main partition maximum capacity (in MiB): 2384185
  Alternate partition maximum capacity (in MiB): 0

While running, mbuffer provides a readout of the buffer input and output speeds:

in @  160 MiB/s, out @  160 MiB/s,  132 GiB total, buffer 100% full


Change the OWNER, TMP_LISTING and NEW_LISTING variables as needed before starting. Note that it's up to you to position the tape at the first file number you want written. The script does not rewind or go to end of tape automatically.

sudo ./ <FILE, ...>

The script will write each argument to a new tar file on the tape, each with a different file number. It's best to point it at logical chunks in directories rather than individual files. The tape position is not altered on start: writing will occur at the currently selected file number (mt -f /dev/nst0 status to check).

When the script ends, the tape position is left at the next available file number. The script can be called multiple times to keep adding items to a tape.

Reading back off the tape

# Position the tape at the desired file number
sudo mt -f /dev/nst0 asf 12

# Extract from the archive
# (Why this uses dd:
sudo dd if=/dev/nst0 bs=1M | tar xvf -

Other notes

  • It's possible to do a partial tar extraction from a position on the tape, but it requires reading from at least the beginning of the archive up to the files you want to extract: the tape can't jump to a file in the tar archive unless you know its byte offset. Due to this it's best to write archives for logical chunks that you are likely to want to restore as a whole (and have the drive space required to do so).

  • /dev/nst0 is used instead of /dev/st0 in all commands as it is the "no rewind" variant of the device. Using /dev/st0 causes an automatic rewind after every operation, which isn't helpful here.

if [[ $(id -u) -ne 0 ]]; then
echo "This script must be run as root"
exit 1
for arg; do
echo '---- Start:' $(date);
mt -f /dev/nst0 status;
mt -f /dev/nst0 status | grep File >> $NEW_LISTING;
sg_logs /dev/nst0 -p tc_;
du "$arg" && tar --index-file=$TMP_LISTING --blocking-factor=512 -cvvf - "$arg" | \
mbuffer -m 10G -L -P 80 | \
dd of=/dev/nst0 bs=256K iflag=fullblock;
echo '---- End:' $(date);
echo '===== End all =====' $(date);
mt -f /dev/nst0 status;
sg_logs /dev/nst0 -p tc_;
echo "File index created: $NEW_LISTING"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment