Skip to content

Instantly share code, notes, and snippets.

@KartikTalwar
Last active November 28, 2024 08:39
Show Gist options
  • Save KartikTalwar/4393116 to your computer and use it in GitHub Desktop.
Save KartikTalwar/4393116 to your computer and use it in GitHub Desktop.
Rsync over SSH - (40MB/s over 1GB NICs)

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

  • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
  • H: preserves hard-links
  • A: preserves ACLs
  • X: preserves extended attributes
  • x: don't cross file-system boundaries
  • v: increase verbosity
  • --numeric-ds: don't map uid/gid values by user/group name
  • --delete: delete extraneous files from dest dirs (differential clean-up during sync)
  • --progress: show progress during transfer

ssh

  • T: turn off pseudo-tty to decrease cpu load on destination.
  • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.
  • o Compression=no: Turn off SSH compression.
  • x: turn off X forwarding if it is on by default.

Original

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

Flip

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>
@danielmotaleite
Copy link

danielmotaleite commented Oct 7, 2019

i have done a small up-to-date ssh test using several cipher, between 2 AWS r5.12xlarge and got this:

chacha20-poly1305@openssh.com        190.89MB/s  (default if not option used)
aes128-ctr                           259.01MB/s
aes256-gcm@openssh.com               339.05MB/s
aes128-gcm@openssh.com               298.20MB/s
none                                 189.72MB/s

i didn't test arcfour, but in previous tests, it was faster... but as require changing the sshd server to support that cipher, i'm trying to avoid it
interesting is the aes256-gcm is faster than the aes128-gcm, probably because of optimization and hardware support. the cipher none, while it do not return error, seems to fallback to the default, so anyone saying that -c none will disable encryption probably do not know that its really using the default cipher! :)

No other ciphers were tested as current ssh only have those as default enabled ciphers

@sasha2002
Copy link

To all.

I'd like to know how to place '--exclude /backup/somedir' in the string because it doesn't work. Thanks in advance.

To exclude a directory("<source_dir>/bigDir") you need to put the name of directory in current directory like this example :
rsync -aHAXxv --numeric-ids --delete --progress --exclude 'bigDir' -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" user@:<source_dir> <dest_dir>

@davidbitton
Copy link

for a Mac to Linux transfer it's useful to use other options. Arcfour is not available for most new machines anymore and UTF-8 on OS X is different than UTF-8 on Linux (important if you have Umlauts like Germans, Samba/NFS will fail otherwise). My command if both (Mac & Linux) machines support AES on their processors and you want to transfer from Mac to Linux:

rsync -rltv --progress --human-readable --delete --iconv=utf-8-mac,utf-8 -e 'ssh -T -c aes128-gcm@openssh.com -o Compression=no -x' <local_mac_source> <remote_linux_dest>

reverse the iconv option if you want to transfer from Linux to Mac.

how would you do mac to mac?

@L1so
Copy link

L1so commented Jun 3, 2020

Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.

@brianlamb
Copy link

This was such a great post to find!
I was set to leave my transfer going at 5-10MB/s but couldn't go to sleep with 1.2TB going for 30hours!
(This was also from Synology NAS to MacOS)

As others mentioned early on in this post using specific SSH options can affect the transfer rate dramatically: -e "ssh -T -c aes128-ctr -o Compression=no -x". Primarily the Compression factor. I couldn't see notable differences and didn't test more than comparing to "-c aes256-gcm@openssh.com" but got variably up to 50-90MB/sec.

DO use --dry-run and --itemize-changes which is a great record of what is actually going to happen.
Always be careful of SOURCE and DESTINATION.
Pause and think, before setting things in motion!

If you want a little help managing a collection of commands you run and an environment conducive to setting up rsync command lines you could try (on a Mac) RsyncOSX as a GUI front end (although I still prefer to run the actual command in a standalone terminal.)

@pricesgoingup
Copy link

@L1so > Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.

Don't paste everything you see on the internet without looking the flags up first /shrugs

@nerrons
Copy link

nerrons commented Mar 29, 2021

To decide which cipher is the best, I recommend using this script to benchmark for yourself: https://gist.github.com/joeharr4/c7599c52f9fad9e53f62e9c8ae690e6b

@anacondaq
Copy link

dunno, dunno
image

git, tons of settings tried. 100gb+ repos. Millions of files, etc of content. Just for bench purposes (not working dirs). But results pretty sad on internal network on Vultr.

@j4ys0n
Copy link

j4ys0n commented May 19, 2022

strange - i'm only getting 27-30MB/s with rsync -aHAxv --numeric-ids --progress -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x"
transferring a 1.6TB file between two Epyc servers with plenty of resources and 10GB networking. i get 500MB/s transferring video files from my desktop to my storage server. not sure what's up here.

update: it increased to 52 MB/s, which is... definitely not as fast as I would like, but it's fine.

@sharkymcdongles
Copy link

Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.

🤡

@ip-rw
Copy link

ip-rw commented Sep 11, 2022

I'm not sure if people are still interested in this, but if you don't care about encryption then tar + netcat is by far the quickest way to transfer directories:

destination:
nc -l -p 7777 | tar -xpf -

source:
tar -cf - sourceDir/ | nc [dest ip] 7777

throw in 'pv' to see xfer speed.

@pricesgoingup
Copy link

pricesgoingup commented Sep 11, 2022 via email

@jaimehrubiks
Copy link

Great discussion.

I found this to be the best option. "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" .Probably aes256 was faster than arcfour due to hardware optimizations or something. Also might play with/without rsync -z based on the quantity/size of the files to transfer. No compression was faster for an already compressed single big tar.gz file

@pricesgoingup
Copy link

pricesgoingup commented Sep 27, 2022 via email

@schmorp
Copy link

schmorp commented Mar 3, 2024

To not let this stand as is, some facts: compression is off by default in ssh (and always has been in openssh), tty allocation is off when used in rsync and x forwarding does not affect bulk bandwidth in any way. Any difference in speed measured is not due to these options, but more likely because of a bad test setup, such as first making tests with cold disk cache and the with hot cache. The only change that can affect speed is the cipher (and not turning compression explicitly on in rsync or ssh).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment