Skip to content

Instantly share code, notes, and snippets.

@DF-wu
Forked from MADscientist314/xargsync.sh
Created July 8, 2023 08:31
Show Gist options
  • Save DF-wu/539e3a320e923e5206626b58eaeb8f5d to your computer and use it in GitHub Desktop.
Save DF-wu/539e3a320e923e5206626b58eaeb8f5d to your computer and use it in GitHub Desktop.
execute a parallel rsync with xargs for large data transfers
#!/bin/bash
# Parallel rsync script originally designed for rsyncing
# large ata transfers from RAID to RAID for the Aagaard Lab.
# Author: Michael Jochum
# Location: Baylor College of Medicine, Houston, TX, USA
# Contact : michael.jochum@bcm.edu
# Date : 2 November 2020
##################################
#Step 0: fill this shit out
##################################
#source directory path wehere you are currently located
SRC=$PWD;
#destination servername@10.ip.address:/path/to/your/dest/dir
DEST="/path/to/dest/dir"
#number of simultaneous rsync connections
#dont be stupid about this or it will be slower than 1 rsync
CON=10
##################################
#Step 1 change into the source dir
##################################
cd $SRC;
##################################
#Step 1 execute rsync with xargs
##################################
#Step 2 #execute this command
ls $SRC |xargs -n1 -P$CON -I% rsync -Pa % $DEST
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment