Instantly share code, notes, and snippets.

Embed
What would you like to do?
Parallel-ise an rsync transfer when you want multiple concurrent transfers happening,
#!/bin/bash
set -e
# Usage:
# rsync_parallel.sh [--parallel=N] [rsync args...]
#
# Options:
# --parallel=N Use N parallel processes for transfer. Defaults to 10.
#
# Notes:
# * Requires GNU Parallel
# * Use with ssh-keys. Lots of password prompts will get very annoying.
# * Does an itemize-changes first, then chunks the resulting file list and launches N parallel
# rsyncs to transfer a chunk each.
# * be a little careful with the options you pass through to rsync. Normal ones will work, you
# might want to test weird options upfront.
#
if [[ "$1" == --parallel=* ]]; then
PARALLEL="${1##*=}"
shift
else
PARALLEL=10
fi
echo "Using up to $PARALLEL processes for transfer..."
TMPDIR=$(mktemp -d)
trap "rm -rf $TMPDIR" EXIT
echo "Figuring out file list..."
# sorted by size (descending)
rsync $@ --out-format="%l %n" --no-v --dry-run | sort -n -r > $TMPDIR/files.all
# check for nothing-to-do
TOTAL_FILES=$(cat $TMPDIR/files.all | wc -l)
if [ "$TOTAL_FILES" -eq "0" ]; then
echo "Nothing to transfer :)"
exit 0
fi
function array_min {
# return the (index, value) of the minimum element in the array
IC=($(tr ' ' '\n' <<<$@ | cat -n | sort -k2,2nr | tail -n1))
echo $((${IC[0]} - 1)) ${IC[1]}
}
echo "Calculating chunks..."
# declare chunk-size array
for ((I = 0 ; I < PARALLEL ; I++ )); do
CHUNKS["$I"]=0
done
# add each file to the emptiest chunk, so they're as balanced by size as possible
while read FSIZE FPATH; do
MIN=($(array_min ${CHUNKS[@]}))
CHUNKS["${MIN[0]}"]=$((${CHUNKS["${MIN[0]}"]} + $FSIZE))
echo $FPATH >> $TMPDIR/chunk.${MIN[0]}
done < $TMPDIR/files.all
find "$TMPDIR" -type f -name "chunk.*" -printf "\n*** %p ***\n" -exec cat {} \;
echo "Starting transfers..."
find "$TMPDIR" -type f -name "chunk.*" | parallel -j $PARALLEL -t --verbose --progress rsync --files-from={} $@
@ben-ba

This comment has been minimized.

Show comment
Hide comment
@ben-ba

ben-ba Dec 11, 2013

Which arguments do you use with with rsync when you call your script?

ben-ba commented Dec 11, 2013

Which arguments do you use with with rsync when you call your script?

@taverentech

This comment has been minimized.

Show comment
Hide comment
@taverentech

taverentech May 4, 2014

This could use a simple check at the beginning to let you know you're missing a dependency instead of spending several minutes finding files and calculating chunks and then failing!

READY=parallel date || ( echo "FATAL: parallel not installed!" && exit 1)

taverentech commented May 4, 2014

This could use a simple check at the beginning to let you know you're missing a dependency instead of spending several minutes finding files and calculating chunks and then failing!

READY=parallel date || ( echo "FATAL: parallel not installed!" && exit 1)

@aladrin

This comment has been minimized.

Show comment
Hide comment
@aladrin

aladrin Jun 11, 2014

which parallel || exit 1

(less complicated the better)

aladrin commented Jun 11, 2014

which parallel || exit 1

(less complicated the better)

@akorn

This comment has been minimized.

Show comment
Hide comment
@akorn

akorn Jul 12, 2014

I rewrote this script in zsh and made it much faster by:

akorn commented Jul 12, 2014

I rewrote this script in zsh and made it much faster by:

@TuHolmes

This comment has been minimized.

Show comment
Hide comment
@TuHolmes

TuHolmes Sep 24, 2014

Fantastic.

I did add the check for the parallel binary, but other than that, great.

I tried the zsh above and had some errors in it when doing the copies, so I decided to use this one.

TuHolmes commented Sep 24, 2014

Fantastic.

I did add the check for the parallel binary, but other than that, great.

I tried the zsh above and had some errors in it when doing the copies, so I decided to use this one.

@NoodlesNZ

This comment has been minimized.

Show comment
Hide comment
@NoodlesNZ

NoodlesNZ Oct 27, 2014

Is there anyway to use this without using "sort -k2,2nr"? I'm trying to run this on a NAS that runs busybox and the sort only supports -n

NoodlesNZ commented Oct 27, 2014

Is there anyway to use this without using "sort -k2,2nr"? I'm trying to run this on a NAS that runs busybox and the sort only supports -n

@rmoorecpcc

This comment has been minimized.

Show comment
Hide comment
@rmoorecpcc

rmoorecpcc May 12, 2015

To increase the speed of the chunk file creation, use the split tool:

cat $TMPDIR/files.all | cut -d\  -f2 | split -d -a1 -n r/10 - $TMPDIR/chunk.

rmoorecpcc commented May 12, 2015

To increase the speed of the chunk file creation, use the split tool:

cat $TMPDIR/files.all | cut -d\  -f2 | split -d -a1 -n r/10 - $TMPDIR/chunk.
@sagary2j

This comment has been minimized.

Show comment
Hide comment
@sagary2j

sagary2j Oct 18, 2015

Hello guys,

I am trying to transfer buch of directories using rsync with the help of multithreading which will start transferring directories and all its sub directories in parallel so that it will reduce the transfer time between two remote servers. can you help me with the code?

sagary2j commented Oct 18, 2015

Hello guys,

I am trying to transfer buch of directories using rsync with the help of multithreading which will start transferring directories and all its sub directories in parallel so that it will reduce the transfer time between two remote servers. can you help me with the code?

@alk224

This comment has been minimized.

Show comment
Hide comment
@alk224

alk224 Aug 20, 2016

As was pointed out in the comments of this related gist, this script is great but the way the files are listed prevents copying of entire directories since a portion of the path for each file is duplicated. Here is an example of a failed output using the above script:

rsync: link_stat "/home/enggen/RAID5/UTK_ITS_test/UTK_ITS_test/strip_primers_out_ITS4_Funrc-5.8S_Funrc_3prime" failed: No such file or directory (2)  
rsync: link_stat "/home/enggen/RAID5/UTK_ITS_test/UTK_ITS_test/strip_primers_out_ITS4_Funrc-5.8S_Funrc_3prime/log_strip_primers_20160803_0640PM.txt" failed: No such file or directory (2)  
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]  
building file list ... 0 files to consider  

There you can see the directory "UTK_ITS_test" is repeated in the file path, and nothing happens.

I changed line 32 on my system (Ubuntu 14.04) thusly and now it works:

rsync $@ --out-format="%l %n" --no-v --dry-run -- | sort -n -r | grep -v "sending incremental file list" | sed -r 's@\s\w+[^/]@ @g' > $TMPDIR/files.all  

I used "@" as delimiter for the sed command since I was matching the first slash which is typically used for delimiting in sed. Please note that sed comes in various flavors and this may not work as well for you. A real coder can probably see a better, more portable solution here.

Edit:
I should point out that this failed to put all the the files transferred in the top-most directory.

Edit2:
Alas, it didn't actually work correctly. Anyone else have an idea?

alk224 commented Aug 20, 2016

As was pointed out in the comments of this related gist, this script is great but the way the files are listed prevents copying of entire directories since a portion of the path for each file is duplicated. Here is an example of a failed output using the above script:

rsync: link_stat "/home/enggen/RAID5/UTK_ITS_test/UTK_ITS_test/strip_primers_out_ITS4_Funrc-5.8S_Funrc_3prime" failed: No such file or directory (2)  
rsync: link_stat "/home/enggen/RAID5/UTK_ITS_test/UTK_ITS_test/strip_primers_out_ITS4_Funrc-5.8S_Funrc_3prime/log_strip_primers_20160803_0640PM.txt" failed: No such file or directory (2)  
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]  
building file list ... 0 files to consider  

There you can see the directory "UTK_ITS_test" is repeated in the file path, and nothing happens.

I changed line 32 on my system (Ubuntu 14.04) thusly and now it works:

rsync $@ --out-format="%l %n" --no-v --dry-run -- | sort -n -r | grep -v "sending incremental file list" | sed -r 's@\s\w+[^/]@ @g' > $TMPDIR/files.all  

I used "@" as delimiter for the sed command since I was matching the first slash which is typically used for delimiting in sed. Please note that sed comes in various flavors and this may not work as well for you. A real coder can probably see a better, more portable solution here.

Edit:
I should point out that this failed to put all the the files transferred in the top-most directory.

Edit2:
Alas, it didn't actually work correctly. Anyone else have an idea?

@sammcj

This comment has been minimized.

Show comment
Hide comment
@sammcj

sammcj Sep 23, 2016

very neat idea, I have an odd bug preventing me from using it on bsd (osx) however:

samm at samm-imac in ~ ~/git/scripts/parallel_rsync.sh -W -r -m --numeric-ids --progress --update --list-only /Volumes/RAID10/music/ /Volumes/bigdata/music/
Using up to 10 processes for transfer...
Figuring out file list...
Calculating chunks...
/Users/samm/git/scripts/parallel_rsync.sh: line 56: 0 + drwx------: syntax error: operand expected (error token is "-")
find: -printf: unknown primary or operator

So it doesn't like something about this on bsd / osx?

sammcj commented Sep 23, 2016

very neat idea, I have an odd bug preventing me from using it on bsd (osx) however:

samm at samm-imac in ~ ~/git/scripts/parallel_rsync.sh -W -r -m --numeric-ids --progress --update --list-only /Volumes/RAID10/music/ /Volumes/bigdata/music/
Using up to 10 processes for transfer...
Figuring out file list...
Calculating chunks...
/Users/samm/git/scripts/parallel_rsync.sh: line 56: 0 + drwx------: syntax error: operand expected (error token is "-")
find: -printf: unknown primary or operator

So it doesn't like something about this on bsd / osx?

@t-animal

This comment has been minimized.

Show comment
Hide comment
@t-animal

t-animal Dec 27, 2016

You should put quotes around $@ like so: "$@" in case a argument contains whitespace.

t-animal commented Dec 27, 2016

You should put quotes around $@ like so: "$@" in case a argument contains whitespace.

@csbogdan

This comment has been minimized.

Show comment
Hide comment
@csbogdan

csbogdan Jul 15, 2017

@rmoorecpcc That works great. For Centos 6 you'll need to manually compile newer coreutils than what default repos provide (which is 8.4). 8.22 works, didn't try older than that.

csbogdan commented Jul 15, 2017

@rmoorecpcc That works great. For Centos 6 you'll need to manually compile newer coreutils than what default repos provide (which is 8.4). 8.22 works, didn't try older than that.

@gerbier

This comment has been minimized.

Show comment
Hide comment
@gerbier

gerbier Jul 21, 2017

I had just a little change to use as many threads as possible :

if [[ "$1" == --parallel=* ]]; then
PARALLEL="${1##*=}"
shift
elif [ -f /proc/cpuinfo ]
then
PARALLEL=$( grep processor /proc/cpuinfo | wc -l )
else
PARALLEL=10
fi

gerbier commented Jul 21, 2017

I had just a little change to use as many threads as possible :

if [[ "$1" == --parallel=* ]]; then
PARALLEL="${1##*=}"
shift
elif [ -f /proc/cpuinfo ]
then
PARALLEL=$( grep processor /proc/cpuinfo | wc -l )
else
PARALLEL=10
fi

@oijm17

This comment has been minimized.

Show comment
Hide comment
@oijm17

oijm17 Oct 5, 2017

I have the next error:
./rsync_parallel.sh --parallel=2 -P -a -h -v -r /DataVolume/shares/BOOTCAMP/mycloud.img /DataVolume/shares/WDMyCloud/
Using up to 1 processes for transfer...
Figuring out file list...
Calculating chunks...

*** /tmp/tmp.1AFWstVhH3/chunk.0 ***
mycloud.img
Starting transfers...
./rsync_parallel.sh: line 63: parallel: command not found

oijm17 commented Oct 5, 2017

I have the next error:
./rsync_parallel.sh --parallel=2 -P -a -h -v -r /DataVolume/shares/BOOTCAMP/mycloud.img /DataVolume/shares/WDMyCloud/
Using up to 1 processes for transfer...
Figuring out file list...
Calculating chunks...

*** /tmp/tmp.1AFWstVhH3/chunk.0 ***
mycloud.img
Starting transfers...
./rsync_parallel.sh: line 63: parallel: command not found

@michaelletzgus

This comment has been minimized.

Show comment
Hide comment
@michaelletzgus

michaelletzgus Jun 26, 2018

Line 57:

echo $FPATH >> $TMPDIR/chunk.${MIN[0]}

must be

echo "$FPATH" >> $TMPDIR/chunk.${MIN[0]}

Otherwise, double spaces in path name will be squeezed.

michaelletzgus commented Jun 26, 2018

Line 57:

echo $FPATH >> $TMPDIR/chunk.${MIN[0]}

must be

echo "$FPATH" >> $TMPDIR/chunk.${MIN[0]}

Otherwise, double spaces in path name will be squeezed.

@michaelletzgus

This comment has been minimized.

Show comment
Hide comment
@michaelletzgus

michaelletzgus Jun 28, 2018

Using splitfor chunk file generation:

cat $TMPDIR/files.all | cut -d" " -f2- | split -d -a3 -n r/$PARALLEL - $TMPDIR/chunk.

This (-f2-) deals with spaces in file names.

michaelletzgus commented Jun 28, 2018

Using splitfor chunk file generation:

cat $TMPDIR/files.all | cut -d" " -f2- | split -d -a3 -n r/$PARALLEL - $TMPDIR/chunk.

This (-f2-) deals with spaces in file names.

@nathanhaigh

This comment has been minimized.

Show comment
Hide comment
@nathanhaigh

nathanhaigh Aug 21, 2018

The algorithm you use for generating CHUNKS which are as balanced as possible in terms of file sizes is good. However, the implementation is slow.

For a list of 74,814 files your while loop took 289s to complete. My implementation was almost 3.5x faster at just 86s and generated chunks of the same size:

function array_min {
  ARR=("$@")

  # Default index for min value
  min_i=0

  # Default min value
  min_v=${ARR[$min_i]}

  for i in "${!ARR[@]}"; do
    v="${ARR[$i]}"

    (( v < min_v )) && min_v=$v && min_i=$i
  done

  echo "${min_i}"
}

The while loop then looks like this (with some progress reporting included:

PROGRESS=0
SECONDS=0
while read FSIZE FPATH; do
  PROGRESS=$((PROGRESS+1))

  # Original Implementation
  #MIN=($(array_min_old ${CHUNKS[@]})); MIN_I=${MIN[0]}
  # Nathan's implementation
  MIN_I=$(array_min ${CHUNKS[@]})

  CHUNKS[${MIN_I}]=$((${CHUNKS[${MIN_I}]} + ${FSIZE}))
  echo "${FPATH}" >> "${TMPDIR}/chunk.${MIN_I}"

  if ! ((PROGRESS % 5000)); then
    >&2 echo "${SECONDS}s: ${PROGRESS} of ${TOTAL_FILES}"
  fi
done < "${TMPDIR}/files.all"
echo "${SECONDS}s"

nathanhaigh commented Aug 21, 2018

The algorithm you use for generating CHUNKS which are as balanced as possible in terms of file sizes is good. However, the implementation is slow.

For a list of 74,814 files your while loop took 289s to complete. My implementation was almost 3.5x faster at just 86s and generated chunks of the same size:

function array_min {
  ARR=("$@")

  # Default index for min value
  min_i=0

  # Default min value
  min_v=${ARR[$min_i]}

  for i in "${!ARR[@]}"; do
    v="${ARR[$i]}"

    (( v < min_v )) && min_v=$v && min_i=$i
  done

  echo "${min_i}"
}

The while loop then looks like this (with some progress reporting included:

PROGRESS=0
SECONDS=0
while read FSIZE FPATH; do
  PROGRESS=$((PROGRESS+1))

  # Original Implementation
  #MIN=($(array_min_old ${CHUNKS[@]})); MIN_I=${MIN[0]}
  # Nathan's implementation
  MIN_I=$(array_min ${CHUNKS[@]})

  CHUNKS[${MIN_I}]=$((${CHUNKS[${MIN_I}]} + ${FSIZE}))
  echo "${FPATH}" >> "${TMPDIR}/chunk.${MIN_I}"

  if ! ((PROGRESS % 5000)); then
    >&2 echo "${SECONDS}s: ${PROGRESS} of ${TOTAL_FILES}"
  fi
done < "${TMPDIR}/files.all"
echo "${SECONDS}s"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment