Skip to content

Instantly share code, notes, and snippets.

@t-cyrill
Last active January 16, 2024 00:23
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save t-cyrill/75e7d90f62d39698f0d18ca4e961ffd3 to your computer and use it in GitHub Desktop.
Save t-cyrill/75e7d90f62d39698f0d18ca4e961ffd3 to your computer and use it in GitHub Desktop.
rsync with pigz
#!/bin/bash
# Usage:
# rsync_pigz $REMOTE:$SRC $DEST
# rsync_pigz $SRC $REMOTE:$DEST
SRC=$1
DEST=$2
if [[ $SRC == *:* ]] && [[ $DEST == *:* ]]; then
echo "Cannot use remote both SRC and DEST"
exit 1
elif [[ $SRC == *:* ]]; then
# REMOTE TO LOCAL
REMOTE=$(echo $SRC | cut -d: -f1)
REMOTE_SRC=$(echo $SRC | cut -d: -f2)
TMPFILE=$(ssh $REMOTE "mktemp")
mkdir -p $DEST
rsync -avn $REMOTE:$REMOTE_SRC $DEST | awk '$0 == "./" { while ($0 != "") { getline; print } exit }' | grep -v '/$' | ssh $REMOTE "cat - > $TMPFILE"
ssh $REMOTE "tar cf - --use-compress-prog=pigz -C $REMOTE_SRC -T $TMPFILE" | tar xzvf - -C $DEST
ssh $REMOTE "rm -f $TMPFILE"
rsync -av $REMOTE:$REMOTE_SRC $DEST
exit $?
elif [[ $DEST == *:* ]]; then
# LOCAL TO REMOTE
REMOTE=$(echo $DEST | cut -d: -f1)
REMOTE_DEST=$(echo $DEST | cut -d: -f2)
TMPFILE=$(mktemp)
ssh $REMOTE "mkdir -p $REMOTE_DEST"
rsync -avn $SRC $REMOTE:$REMOTE_DEST | awk '$0 == "./" { while ($0 != "") { getline; print } exit }' | grep -v '/$' > $TMPFILE
tar cvf - --use-compress-prog=pigz -C $SRC -T $TMPFILE | ssh $REMOTE "tar xzf - -C $REMOTE_DEST"
rm -f $TMPFILE
rsync -av $SRC $REMOTE:$REMOTE_DEST
exit $?
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment