Skip to content

Instantly share code, notes, and snippets.

@v6ak
Last active August 29, 2015 14:10
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save v6ak/ec486fae2e03db75ae35 to your computer and use it in GitHub Desktop.
Save v6ak/ec486fae2e03db75ae35 to your computer and use it in GitHub Desktop.
Don't want to wait for computations (e.g. checksums) on large files after they are downloaded? Do the computation during the download!
#!/bin/bash
# No warranty is provided!
#
# Rather hacky script for streaming downloaded files through pipe.
# Requires inotify-tools package installed.
# It does not handle some failures properly!
#
# Assumes that a browser saves the temporary result into the .part file and the browser is the only process that opens the file for writing.
#
# How it works: The script streams the content of the .part file to stdout. When the .part file is closed, it stops streaming.
#
# Note that I am not sure if there is a race condition that could stop the streaming prematurely in some cases. Some simple experiments suggest that it can't happen, but I am not 100% sure.
#
# Example of usage: catdld some-large-file-being-downloaded | sha256sum
# This will compute the sha256 of a large file in the background while the file is being downloaded, so you don't have to wait for checking the checksum after download. Because the CPU is much faster than the network, the checksum is available virtually immediately after the download. It may also save disk reads thanks to caches.
FILE="$1"
PART="$FILE.part"
inotifywait -e close_write "$PART" > /dev/null&
watchpid="$!"
tail -f -c +0 --pid="$watchpid" "$PART" || exit 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment