Skip to content

Instantly share code, notes, and snippets.

@MADscientist314
Created November 2, 2020 17:08
Show Gist options
  • Star 9 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save MADscientist314/a892e2b50aa7a828f081d88e0d1ff393 to your computer and use it in GitHub Desktop.
Save MADscientist314/a892e2b50aa7a828f081d88e0d1ff393 to your computer and use it in GitHub Desktop.
execute a parallel rsync with xargs for large data transfers
#!/bin/bash
# Parallel rsync script originally designed for rsyncing
# large ata transfers from RAID to RAID for the Aagaard Lab.
# Author: Michael Jochum
# Location: Baylor College of Medicine, Houston, TX, USA
# Contact : michael.jochum@bcm.edu
# Date : 2 November 2020
##################################
#Step 0: fill this shit out
##################################
#source directory path wehere you are currently located
SRC=$PWD;
#destination servername@10.ip.address:/path/to/your/dest/dir
DEST="/path/to/dest/dir"
#number of simultaneous rsync connections
#dont be stupid about this or it will be slower than 1 rsync
CON=10
##################################
#Step 1 change into the source dir
##################################
cd $SRC;
##################################
#Step 1 execute rsync with xargs
##################################
#Step 2 #execute this command
ls $SRC |xargs -n1 -P$CON -I% rsync -Pa % $DEST
@J-Massey
Copy link

J-Massey commented Sep 6, 2021

Is there an elegant solution to transferring files from the remote server to the local machine?

@MADscientist314
Copy link
Author

I suppose the easiest solution would be to execute this script via ssh on the remote server and make the destination the local machine

@J-Massey
Copy link

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment