Skip to content

Instantly share code, notes, and snippets.

@oliverisaac
Created January 28, 2025 19:40
Show Gist options
  • Save oliverisaac/31cc6b037e1b408d415373343deb1a22 to your computer and use it in GitHub Desktop.
Save oliverisaac/31cc6b037e1b408d415373343deb1a22 to your computer and use it in GitHub Desktop.
From Overnight to Over Lunch: Moving Terabytes of Data At 10 Gigabits per Second
rsync -avh -e "ssh -T -o Compression=no -x -c aes128-gcm@openssh.com" /var/lib/mysql/ destination:/var/lib/mysql
#!/usr/bin/env bash
IFS=$'\n'
num_parallel=10
cd /var/lib/mysql
file_list_dir=$( mktemp -d )
all_files=( $( find . -type f -print ) )
i=0
for f in "${all_files[@]}"; do
index=$(( i % num_parallel ))
echo "$f" >> "${file_list_dir}/$index.lst"
i=$(( i + 1 ))
done
for (( i=0; i<num_parallel; i++)); do
rsync -avh -e "ssh -T -o Compression=no -x -c aes128-gcm@openssh.com" --files-from="${file_list_dir}/$i.lst" "$PWD/" "destination:$PWD/" &
done
wait
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment