Instantly share code, notes, and snippets.

@KartikTalwar KartikTalwar/Documentation.md
Last active Feb 22, 2019

Embed
What would you like to do?
Download ZIP
Rsync over SSH - (40MB/s over 1GB NICs)
Raw
Documentation.md

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

  • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
  • H: preserves hard-links
  • A: preserves ACLs
  • X: preserves extended attributes
  • x: don't cross file-system boundaries
  • v: increase verbosity
  • --numeric-ds: don't map uid/gid values by user/group name
  • --delete: delete extraneous files from dest dirs (differential clean-up during sync)
  • --progress: show progress during transfer

ssh

  • T: turn off pseudo-tty to decrease cpu load on destination.
  • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.
  • o Compression=no: Turn off SSH compression.
  • x: turn off X forwarding if it is on by default.

Original

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

Flip

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]
Raw
SSHrsync.sh
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>
@hollodk

This comment has been minimized.

Sign in to view
Copy link

hollodk commented Nov 20, 2013

thanks.. you just saved me a few hours.
@reox

This comment has been minimized.

Sign in to view
Copy link

reox commented Dec 25, 2013

nice, got up to 64MB/s over Gbit Network.
faster is only the direct netcat approach (about 80MB/s):
on the sending side: tar -c /path/to/data | nc -q 10 12345
on the receiving side: nc -w 10 -l -p 12345 | tar -x
but thats obviously unencrypted.
@cleytonferrari

This comment has been minimized.

Sign in to view
Copy link

cleytonferrari commented Mar 19, 2014

Muito Obrigado!
@TheMengzor

This comment has been minimized.

Sign in to view
Copy link

TheMengzor commented Jul 6, 2014

MAAAAAAAAD!
Thank you so much!
@reox

This comment has been minimized.

Sign in to view
Copy link

reox commented Oct 12, 2014

by the way: i could observe 110MB/s with intel NICs and SSD to SSD transfer. So the real bottleneck is only your harddisk
@gozoinks

This comment has been minimized.

Sign in to view
Copy link

gozoinks commented Jan 25, 2015

Hard disk performance is not remotely the only bottleneck. Encryption and compression algorithms have wildly varying performance as well. All else being equal, SSD or no, using faster/lighter encryption and faster or no compression can improve payload throughput considerably. And it has as much to do with the material being transferred as anything else, particularly if the source is difficult to compress or encrypt. These things can't be trivialized, and careful attention to SSH options can be very important for performance.

Thanks to the OP. T c o and x are valuable options to use for bulk transfers.
@denji

This comment has been minimized.

Sign in to view
Copy link

denji commented Mar 2, 2015 

ciphers arcfour256,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
@bubenkoff

This comment has been minimized.

Sign in to view
Copy link

bubenkoff commented Mar 8, 2015

thanks a lot for this gist!
@priyadarshan

This comment has been minimized.

Sign in to view
Copy link

priyadarshan commented Mar 20, 2015

If you use the rsync:// protocol scheme (i.e. when you connect to a rsyncd daemon) then no encryption will be used (although password authentication is done using a MD4-based challenge-response system and is probably still reasonably secure). See: http://unix.stackexchange.com/questions/191249/understanding-why-such-a-discrepancy-in-network-transfer
@rickw

This comment has been minimized.

Sign in to view
Copy link

rickw commented May 28, 2015

Thanks, MacPro to Ubuntu Box is going at 100+MB/s
@KevinMKorb

This comment has been minimized.

Sign in to view
Copy link

KevinMKorb commented Jul 22, 2015

I understand the desire for using a weak cipher on an internal network but at least use arcfour128. It has essentially the same performance as arcfour (both are 128-bit) but the random session key will actually be random rather than guessable.
@mmitchell816

This comment has been minimized.

Sign in to view
Copy link

mmitchell816 commented Aug 5, 2015

Thanks for this. I was only using vaHz flags and was getting about 15MB between two server each with 8 HDDs, now I'm hitting 100MB+. This just saved me days off my transfer. And thats without changing sshd_config on the remote server. Not that it would help at this point since I'm nearly maxing out the Gig link.
@gpmcadamold

This comment has been minimized.

Sign in to view
Copy link

gpmcadamold commented Aug 12, 2015

@KartikTalwar This is a life-saver, thanks. However, there's a typo in the description:

--numeric-ds

Should be:

--numeric-ids
@2manydjs

This comment has been minimized.

Sign in to view
Copy link

2manydjs commented Aug 26, 2015

Unfortunately I did not get the same improvement between my OS X box and Synology NAS. In specific -e "ssh -c arcfour -o Compression=no" will make it slow down from 17MB/s to 13MB/s (on Gbit).
@kedare

This comment has been minimized.

Sign in to view
Copy link

kedare commented Aug 30, 2015

Got 80MB/s between OS X and a Proliant Microserver n40l :)
@kroko23

This comment has been minimized.

Sign in to view
Copy link

kroko23 commented Dec 13, 2015

I think is faster to mount remote server with nfs and do rsync locally.
@AntonMay

This comment has been minimized.

Sign in to view
Copy link

AntonMay commented Jan 5, 2016

41,200,648,192 100% 94.87MB/s 0:06:54

Awesome!!!!
@minhaz1

This comment has been minimized.

Sign in to view
Copy link

minhaz1 commented Jan 10, 2016

I was able to get around 100 MB/s running rsync in daemon mode in parallel w/ 4 processes, transferring around 1.5 million 800kb files. I'm confident that the 1 Gbps nic was the bottleneck since we have really fast Virident SSD's. With 10-gig nic I probably could get several hundred MB/s.
@aflyhorse

This comment has been minimized.

Sign in to view
Copy link

aflyhorse commented Feb 25, 2016

Since OpenSSH 6.7, the "arcfour" cipher was disabled by default. For example, the sshd ship with Debian Jessie:

ssh -c arcfour sso
Unable to negotiate with x.x.x.x: no matching cipher found. Their offer: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com

For newer hardware (both sides have CPU with AES-NI), I recommend "aes128-gcm@openssh.com" as an alternative.
@kosir

This comment has been minimized.

Sign in to view
Copy link

kosir commented Mar 10, 2016

Kudos! Works perfectly, super fast! 👍
@kosir

This comment has been minimized.

Sign in to view
Copy link

kosir commented Mar 10, 2016

How to exclude (for example) hidden directories? Anything I tried doesn't work.
@Fossil01

This comment has been minimized.

Sign in to view
Copy link

Fossil01 commented May 18, 2016

Sped my transfer up from ~10 MB/s to 30 MB/s from SATA RAID-1 to SATA RAID-1. Will try later with SSD to SSD.
@mautematico

This comment has been minimized.

Sign in to view
Copy link

mautematico commented Jun 10, 2016

sent 1,078 bytes received 167,802 bytes 48,251.43 bytes/sec, over Internet.

My server does not support arcfour, so I removed that option. :)
@stefancrain

This comment has been minimized.

Sign in to view
Copy link

stefancrain commented Jul 19, 2016 •
edited

Great notes @KartikTalwar! I skip compression on filetypes that I know are encrypted and not worth the CPU time to compress. As a result transfers move much fast on my machines.

export RSYNC_SKIP_COMPRESS=3fr/3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/arw/asf/avi/bz2/cab/cr2/crypt[5678]/dat/dcr/deb/dmg/drc/ear/erf/flac/flv/gif/gpg/gz/iiq/iso/jar/jp2/jpeg/jpg/k25/kdc/lz/lzma/lzo/m4[apv]/mef/mkv/mos/mov/mp[34]/mpeg/mp[gv]/msi/nef/oga/ogg/ogv/opus/orf/pef/png/qt/rar/rpm/rw2/rzip/s7z/sfx/sr2/srf/svgz/t[gb]z/tlz/txz/vob/wim/wma/wmv/xz/zip
rsync --skip-compress=$RSYNC_SKIP_COMPRESS .....

Cheers!
@RedShift1

This comment has been minimized.

Sign in to view
Copy link

RedShift1 commented Jul 30, 2016

Nice
@gabrielfsousa

This comment has been minimized.

Sign in to view
Copy link

gabrielfsousa commented Aug 17, 2016

thanks a lot
@rorymcdaniel

This comment has been minimized.

Sign in to view
Copy link

rorymcdaniel commented Sep 5, 2016

Wow, thanks for sharing these, I can't believe how much faster it is.
@therealmarv

This comment has been minimized.

Sign in to view
Copy link

therealmarv commented Sep 16, 2016

for a Mac to Linux transfer it's useful to use other options. Arcfour is not available for most new machines anymore and UTF-8 on OS X is different than UTF-8 on Linux (important if you have Umlauts like Germans, Samba/NFS will fail otherwise). My command if both (Mac & Linux) machines support AES on their processors and you want to transfer from Mac to Linux:

rsync -rltv --progress --human-readable --delete --iconv=utf-8-mac,utf-8 -e 'ssh -T -c aes128-gcm@openssh.com -o Compression=no -x' <local_mac_source> <remote_linux_dest>

reverse the iconv option if you want to transfer from Linux to Mac.
@klein0r

This comment has been minimized.

Sign in to view
Copy link

klein0r commented Oct 26, 2016

Well done! Thanks a lot :) Speeds up a copy process from Synology NAS to NAS a lot!
@HaniKali

This comment has been minimized.

Sign in to view
Copy link

HaniKali commented Dec 15, 2016

Mac-mini:~ hani2$ rsync -a -v --remove-source-files --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" /Users/hani2/Desktop/3TB/Stand\ Up\ Comedy/ /Users/hani2/Desktop/DOCO+STANDUP/Stand\ Up\ Comedy/
building file list ...
188 files to consider
deleting def comedy jams/def.comedy.jam.all.stars.vol.09.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.06.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.05.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.04.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.03.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.02.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.01.avi
deleting def comedy jams/NFO.nfo
deleting def comedy jams/Def Comedy Jam All. Stars vol.08.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.12.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.11.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.10.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.07.avi
deleting def comedy jams/
deleting Louis CK/Louis C.K. Interview.flv
deleting Louis CK/Louis C.K. 1995.flv
deleting Louis CK/Louis C.K. - Shameless.avi
deleting Louis CK/
deleting Lisa Lampanelli/Lisa Lampanelli - Long Live the Queen.avi
deleting Lisa Lampanelli/
deleting Lewis Black/MP3/The White Album/Lewis Black - 12 - Other Idiots from Arkansas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 11 - Education in Arkansas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 10 - The Impeachment.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 09 - Bill Clinton and Oral Sex.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 08 - Heaven's Gate.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 07 - IHOP (The International House of Pancakes).mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 06 - TV Pilot.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 05 - Fuck, New York, Los Angeles.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 04 - Miami and Las Vegas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 03 - The Ozone, Sunblock, The Flu, and Nyquil.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 02 - The Fall, Hurricanes, and Weathermen.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 01 - Wisconsin.mp3
deleting Lewis Black/MP3/The White Album/
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 13 - A Sense of Humor.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 12 - The War in Iraq.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 11 - Bringing Democracy to Iraq.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 10 - Homeland Security.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 09 - Who's Fucking Who.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 08 - Greed.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 07 - Small Pox.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 06 - Health Clubs.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 05 - Ireland & Health.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 04 - International Travel.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 03 - The Coldest Winter Ever.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 02 - The Settling of Minnesota.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 01 - Minnesota Winters.mp3
deleting Lewis Black/MP3/Rules of Enragement/
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 10 - What I've Learned.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 09 - TV News & Jerry Falwell.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 08 - Airport Security.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 07 - Tax Rebates and Common Sense.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 06 - The End of the Universe.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 05 - Halftime At the Super Bowl 200.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 04 - Coke and Pepsi and the Super B.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 03 - Environmental Terrorism or Glo.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 02 - After September 11th.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 01 - Atlanta.mp3
deleting Lewis Black/MP3/End of the Universe/
deleting Lewis Black/MP3/
deleting Lewis Black/Lewis Black on Brodway.avi
deleting Lewis Black/Lewis Black - Taxed Beyond Belief.avi
deleting Lewis Black/Lewis Black - Red White and Screwed.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black 2002.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black 2000.avi
deleting Lewis Black/
deleting Lee Evans/Big Live cd2 [2008].avi
deleting Lee Evans/Big Live cd1 [2008].avi
deleting Lee Evans/
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 5).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 4).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 3).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 2).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 1).avi
deleting Laughing Out Loud/
deleting Latin Divas of Comedy/Latin Divas.divx
deleting Latin Divas of Comedy/
deleting Laffapalooza/Laffapalooza 3 [2007].avi
deleting Laffapalooza/Laffapalooza 2 [2007].avi
deleting Laffapalooza/Laffapalooza 1 [2006].avi
deleting Laffapalooza/
deleting Kyle Cease/Kyle Cease - Weirder Blacker Dimpler [2007].avi
deleting Kyle Cease/
deleting Kings of Comedy/The Original Kings Of Comedy.avi
deleting Kings of Comedy/
deleting Kevin James/Kevin James, sweat the small stuff.mpg
deleting Kevin James/
deleting Katt Williams/Katt Williams - Live [2006].avi
deleting Katt Williams/Katt Williams - Its Pimpin Pimpin [2008].avi
deleting Katt Williams/
deleting Just for Laughs/Just for Laughs 2004 Montreal Comedy Festival.avi
deleting Just for Laughs/
deleting Joey Coco Diaz/Joey Coco Diaz Socially Unacceptable 2016 720p SESO WEB.mp4
deleting Joey Coco Diaz/
deleting Jo Koy/Dont Make Him Angry [2009].avi
deleting Jo Koy/
deleting Jimmy Carr/Jimmy_Carr_--Comedy_Central_stand_up--_Part_2.avi
deleting Jimmy Carr/Jimmy Carr - Stand Up.avi
deleting Jimmy Carr/Jimmy Carr - Live.avi
deleting Jimmy Carr/Jimmy Carr - Live at the Apollo (2).avi
deleting Jimmy Carr/Jimmy Carr - Live at the Apollo (1).avi
deleting Jimmy Carr/Jimmy Carr - In Concert.avi
deleting Jimmy Carr/Jimmy Carr - Comedian .avi
deleting Jimmy Carr/
deleting Jim Gaffigan/Jim Gaffigan - King Baby.avi
deleting Jim Gaffigan/Jim Gaffigan - Beyond The Pale.avi
deleting Jim Gaffigan/
deleting Jim Carrey/Jim Carrey - Unnatural Act (1991).avi
deleting Jim Carrey/Jim Carrey - Standup Routine - Comedy Store (1996).mpg
deleting Jim Carrey/
deleting Jim Breuer/Jim Breuer - Hardcore.avi
deleting Jim Breuer/
deleting Jerry Seinfeld/Stand up/tonight show with micheal richards.avi
deleting Jerry Seinfeld/Stand up/tonight show appearance 1990.avi
deleting Jerry Seinfeld/Stand up/stand up 1981.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 4.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 3.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 2.avi
deleting Jerry Seinfeld/Stand up/exclusive stand up 1.avi
deleting Jerry Seinfeld/Stand up/Saturday Night Live Hosted by Jerry Seinfeld.avi
deleting Jerry Seinfeld/Stand up/
deleting Jerry Seinfeld/Seinfeld Reunion Special.avi
deleting Jerry Seinfeld/Jerry Seinfeld - The Comedian.avi
deleting Jerry Seinfeld/Jerry Seinfeld - Stand-Up Confidential 1987.avi
deleting Jerry Seinfeld/Jerry Seinfeld - Not That There's Anything Wrong With That (Bloopers From S2 DVD).avi
deleting Jerry Seinfeld/Jerry Seinfeld - I`m telling it for the last time.avi
deleting Jerry Seinfeld/
deleting Jeff Dunham/Jeff Dunhams - Very Special Christmas Special [2008].avi
deleting Jeff Dunham/Jeff Dunham - Spark of Insanity [2007].avi
deleting Jeff Dunham/Jeff Dunham - Arguing with Myself [2006].avi
deleting Jeff Dunham/
deleting Jason Byrne/Jason Byrne - The Byrne Identity [2008].avi
deleting Jason Byrne/
deleting Heckler/aaf-heckler.nfo
deleting Heckler/aaf-heckler.avi
deleting Heckler/
deleting George Carlin/George Carlin - The Mark Twain prize.avi
deleting George Carlin/George Carlin - Its bad for ya [2008].avi
deleting George Carlin/George Carlin - Inside The Actor's Studio [2004].avi
deleting George Carlin/George Carlin - George's Best Stuff [1996].avi
deleting George Carlin/George Carlin - 12 - Complaints and Grievances [2001].avi
deleting George Carlin/George Carlin - 11 - You Are All Diseased [1999].avi
deleting George Carlin/George Carlin - 10 - 40 Years Of Comedy [1997].avi
deleting George Carlin/George Carlin - 09 - Back In Town [1996].avi
deleting George Carlin/George Carlin - 08 - Jammin' In New York [1992].avi
deleting George Carlin/George Carlin - 07 - Doin' It Again [1990].avi
deleting George Carlin/George Carlin - 06 - What the Fuck am I Doing in New Jersey [1988].avi
deleting George Carlin/George Carlin - 05 - Playing With Your Head [1986].avi
deleting George Carlin/George Carlin - 04 - Carlin On Campus [1984].avi
deleting George Carlin/George Carlin - 02 - Again [1978].avi
deleting George Carlin/George Carlin - 01 - On Location At Usc [1977].avi
deleting George Carlin/
deleting Gabriel Iglesias/Gabriel Iglesias - Hot and Flufy.divx
deleting Gabriel Iglesias/Comedy Central Presents - Gabriel Iglesias.avi
deleting Gabriel Iglesias/
deleting Frankie Boyle/Frankie Boyle - Live [2008].avi
deleting Frankie Boyle/
deleting Frank Skinner/frank.skinner.standup.live.from.birminghams.nia.dvdrip.xvid-nosegment.nfo
deleting Frank Skinner/frank skinner - live from birminghams nia.avi
deleting Frank Skinner/
deleting Frank Caliendo/Frank Caliendo-Giggles.avi
deleting Frank Caliendo/
deleting Flight Of The Conchords/Flight Of The Conchords - HBO One Night Stand.avi
deleting Flight Of The Conchords/
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (7).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (6).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (5).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (4).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (3).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (2).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (1).avi
deleting Ellen DeGeneres/
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/18 - Eddie Murphy - Seeing Birth - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/17 - Eddie Murphy - Almost Fucked a Midget - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/16 - Eddie Murphy - Black Sambo - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/15 - Eddie Murphy - Little Richard - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/14 - Eddie Murphy - Lost in Space - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/13 - Eddie Murphy - Moses - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/12 - Eddie Murphy - Niggaz in the '70s - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/11 - Eddie Murphy - Buckwheat - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/10 - Eddie Murphy - Grandma Klump - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/09 - Eddie Murphy - Skeleton in Closet - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/08 - Eddie Murphy - Cumin' Hard - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/07 - Eddie Murphy - Hit by a Car - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/06 - Eddie Murphy - Boxers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/05 - Eddie Murphy - Ice Cream Man - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/04 - Eddie Murphy - Old Jew - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/03 - Eddie Murphy - Singers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/02 - Eddie Murphy - Drinking Fathers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/01 - Eddie Murphy - The Barbecue - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/
deleting Eddie Murphy/Eddie Murphy - Raw (1987).avi
deleting Eddie Murphy/Eddie Murphy - Delirious (1983).avi
deleting Eddie Murphy/
deleting Eddie Izzard/Eddie Izzard - Definite Article.avi
deleting Eddie Izzard/
deleting Eddie Griffin/Eddie Griffin - Last Stand [2003].avi
deleting Eddie Griffin/Eddie Griffin - Freedom Of Speech [2008].avi
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (4).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (3).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (2).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (1).flv
deleting Eddie Griffin/.DS_Store
deleting Eddie Griffin/
deleting Dylan Moran/Dylan Moran - Monster.avi
deleting Dylan Moran/Dylan Moran - Melbourne Comedy Gala 2006.avi
deleting Dylan Moran/Dylan Moran - Like Totally.avi
deleting Dylan Moran/
deleting Drew Hastings/fico-dhiam.sfv
deleting Drew Hastings/fico-dhiam.nfo
deleting Drew Hastings/Drew Hastings - Irked and Miffed 2008.avi
deleting Drew Hastings/
deleting Dennis Miller/Dennis Miller The Raw Feed 2003.avi
deleting Dennis Miller/
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 06 Driving home that day.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 05 Someone Tell My Mom.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 04 I want all Americans.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 03 The Asshole Olympics.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 02 Number One.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 01 The Pope is a Pimp.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 12 One thing that drives.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 11 My wife screams.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 10 The big chunk.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 09 We all know women.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 08 We couldn't care less.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 07 This Is Your Brain.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 06 Does This Bomb.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 05 She loves us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 04 Semen count.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 03 I was ready to steamroll.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 02 There was an author.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 01 Grande Vente Mocha.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 12 Three words for all the.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 11 I think its fair to say.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 10 When I walked into the.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 09 Matt Dillon Is A Giant.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 08 Nuns, Tits, Booze.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 07 We decided not to put.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 06 How I Became A.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 05 How I Learned To Act.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 04 And that was the end.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 03 My advice to men.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 02 Now I know how.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 01 Autism Shmautism.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 12 Why the fuck do old.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 11 As he drew closer.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 10 Within seconds Bobby.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 09 Bullies R Us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 08 We never went hungry.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 07 I got a job in a diner.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 06 And on and on it went.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 05 I have definite and vitall.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 04 I Had Sex With.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 03 Please Drug Your Children.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 02 Stabbing screaming.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 01 Your Kids Are Not Cute.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 18 Me.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 17 Angelina Jolie.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 16 Rehab.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 15 NASCAR.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 14 The Double Quarter Pounder.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 13 Bill Clinton Blow Jobs.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 12 You.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 11 Britney Spears's Head.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 10 Britney Spears's Vagina.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 09 George Bush Jr..mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 08 Anna Nicole Smith.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 07 Why Everyone Hates Us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 06 Yup-there is a real.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 05 Danica Patrick.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 04 Okay, this is the prolouge.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 03 A Second Note.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 02 Secret Hidden Message.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 01 Penguin Audio presents.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/
deleting Denis Leary/MP3/Why We Suck/
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 10 Voices in My Head.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 09 Traditional Irish Folk Song.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 08 The Downtrodden Song.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 07 Death.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 06 Meat.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 05 Smoke.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 04 More Drugs.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 03 Rehab.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 02 Drugs.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 01 Asshole.mp3
deleting Denis Leary/MP3/No Cure for Cancer/
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 20 Lock 'n Load.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 19 Fuck The Pope.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 18 Life's Gonna Suck.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 17 My Kids.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 16 Insane cowboy (In Africa).mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 15 Fat Fucks.MP3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 14 Love Barge.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 13 A Reading from the Book of Apple.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 12 President Leary.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 11 Fuck The Kennedys.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 10 I'm Happy.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 09 Elvis and I.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 08 Fuck Santa.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 07 Beer.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 06 Coffee.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 05 Deaf Mute Coctail Party.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 04 Save This.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 03 Marv Marv Marv.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 02 Asshole of the Dance.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 01 Fuck This.mp3
deleting Denis Leary/MP3/Lock 'n Load/
deleting Denis Leary/MP3/
deleting Denis Leary/Denis Leary - No Cure For Cancer.avi
deleting Denis Leary/Denis Leary - Lock 'N Load.avi
deleting Denis Leary/
deleting Demetri Martin/Demetri Martin - If I (6).avi
deleting Demetri Martin/Demetri Martin - If I (5).avi
deleting Demetri Martin/Demetri Martin - If I (4).avi
deleting Demetri Martin/Demetri Martin - If I (3).avi
deleting Demetri Martin/Demetri Martin - If I (2).avi
deleting Demetri Martin/Demetri Martin - If I (1).avi
deleting Demetri Martin/Comedy Central Presents - Demetri Martin.mpg
deleting Demetri Martin/
deleting Dave Chappelle/Dave Chappelle - Killing Them Softly [2003].avi
deleting Dave Chappelle/Dave Chappelle - For what its worth [2004].divx
deleting Dave Chappelle/Dave Chapelle in San Francisco (3).avi
deleting Dave Chappelle/Dave Chapelle in San Francisco (2).avi
deleting Dave Chappelle/Dave Chapelle in San Francisco (1).avi
deleting Dave Chappelle/.DS_Store
deleting Dave Chappelle/
deleting Dave Attell/Dave Attell - Captain Miserable.avi
deleting Dave Attell/Dave Attel - Insomniac Tour [2005].avi
deleting Dave Attell/Comedy Central Presents - Dave Attell.avi
deleting Dave Attell/
deleting Dara O Brian/Dara O Brian - Talks Funny Live In London [2007].avi
deleting Dara O Brian/Dara O Briain - Live at the Theatre Royal [2006].avi
deleting Dara O Brian/
deleting Daniel Tosh/Daniel Tosh - Completely Serious.divx
deleting Daniel Tosh/DVDCover.jpg
deleting Daniel Tosh/
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (5).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (4).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (3).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (2).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (1).avi
deleting Dane Cook/Insomniac Tour/
deleting Dane Cook/rough around the edges.avi
deleting Dane Cook/Stand up Comedy - Dane Cook on Kilborn - Lions.avi
deleting Dane Cook/Dane.Cook.Isolated.Incident.2009.DVDRip.XviD-Delta9.avi
deleting Dane Cook/DANEgerous Edition (iPod-ready).mp4
deleting Dane Cook/Comedy Central Presents - Dane Cook.mpg
deleting Dane Cook/Comedy Central Presents - Dane Cook, Harmful If Swallowed.avi
deleting Dane Cook/
deleting Dana Carvey/Dana Carvey - Critics' Choice (1995).avi
deleting Dana Carvey/
deleting Craig Ferguson/Craig Ferguson a wee bit o revolution 2009.avi
deleting Craig Ferguson/
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 5, 20-1).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 4, 40-21).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 3, 60-41).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 2, 80-61).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 1, 100-81).avi
deleting Comedy central 100 greatest stand ups/
deleting Comedy Central Roasts/The N.Y. Friars Club Roast of Drew Carey.avi
deleting Comedy Central Roasts/The N.Y. Friars Club Roast Of Hugh Hefner.mpg
deleting Comedy Central Roasts/The N.Y. Friars Club Roast Of Chevy Chase.avi
deleting Comedy Central Roasts/Comedy Central Roast of Pamela Anderson - Secret Stash Version.avi
deleting Comedy Central Roasts/Comedy Central Roast of Comedian Jeff Foxworthy.avi
deleting Comedy Central Roasts/Comedy Central Roast of Comedian Denis Leary.avi
deleting Comedy Central Roasts/Comedy Central Roast of Bob Saget.avi
deleting Comedy Central Roasts/Comedy Central Roast Of William Shatner 2006.avi
deleting Comedy Central Roasts/Comedy Central Roast Of Flavor Flav.avi
deleting Comedy Central Roasts/
deleting Comedy Central Presents/Comedy Central Presents - Zach Galifianakis.avi
deleting Comedy Central Presents/Comedy Central Presents - Wanda Sykes.avi
deleting Comedy Central Presents/Comedy Central Presents - Vince Morris.avi
deleting Comedy Central Presents/Comedy Central Presents - Vic Henley.avi
deleting Comedy Central Presents/Comedy Central Presents - Vanessa Hollingshead.avi
deleting Comedy Central Presents/Comedy Central Presents - Ty Barnett.avi
deleting Comedy Central Presents/Comedy Central Presents - Tracy Smith.avi
deleting Comedy Central Presents/Comedy Central Presents - Tony Woods.avi
deleting Comedy Central Presents/Comedy Central Presents - Tommy Johnagin.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Shillue.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Rhodes.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Papa.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Papa(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Cotter.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Lynn.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Glass.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Barry.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Barry(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tig.avi
deleting Comedy Central Presents/Comedy Central Presents - The Sklar Brothers.mpg
deleting Comedy Central Presents/Comedy Central Presents - Tess.avi
deleting Comedy Central Presents/Comedy Central Presents - Ted Alexandro.avi
deleting Comedy Central Presents/Comedy Central Presents - Ted Alexandro(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tammy Pescatelli.avi
deleting Comedy Central Presents/Comedy Central Presents - Sue Murphy.avi
deleting Comedy Central Presents/Comedy Central Presents - Steve McGrew.avi
deleting Comedy Central Presents/Comedy Central Presents - Steve Byrne.avi
deleting Comedy Central Presents/Comedy Central Presents - Stella.avi
deleting Comedy Central Presents/Comedy Central Presents - Slovin & Allen.avi
deleting Comedy Central Presents/Comedy Central Presents - Sheryl Underwood.avi
deleting Comedy Central Presents/Comedy Central Presents - Sebastian Maniscalco.avi
deleting Comedy Central Presents/Comedy Central Presents - Sean Cullen.wmv
deleting Comedy Central Presents/Comedy Central Presents - Scott Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Sabrina Matthews.avi
deleting Comedy Central Presents/Comedy Central Presents - Russell Peters.avi
deleting Comedy Central Presents/Comedy Central Presents - Russ Meneve.avi
deleting Comedy Central Presents/Comedy Central Presents - Rudy Rush.avi
deleting Comedy Central Presents/Comedy Central Presents - Rod Man.avi
deleting Comedy Central Presents/Comedy Central Presents - Rocky LaPorte.avi
deleting Comedy Central Presents/Comedy Central Presents - Robert Kelly.avi
deleting Comedy Central Presents/Comedy Central Presents - Robert Hawkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Rickey Smiley.avi
deleting Comedy Central Presents/Comedy Central Presents - Richard Jeni.avi
deleting Comedy Central Presents/Comedy Central Presents - Rich Vos.avi
deleting Comedy Central Presents/Comedy Central Presents - Retta.avi
deleting Comedy Central Presents/Comedy Central Presents - Reno Collier.avi
deleting Comedy Central Presents/Comedy Central Presents - Rene Hicks.avi
deleting Comedy Central Presents/Comedy Central Presents - Reggie McFadden.avi
deleting Comedy Central Presents/Comedy Central Presents - Ralph Harris.avi
deleting Comedy Central Presents/Comedy Central Presents - Peter Berman.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul Mercurio.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul Gilmartin.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul F.Tompkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul F. Tompkins(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Patrice O'Neal.avi
deleting Comedy Central Presents/Comedy Central Presents - Pat Dixon.avi
deleting Comedy Central Presents/Comedy Central Presents - Otis Lee Crenshaw.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Swardson.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Swardson(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Dipaolo.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick DiPaolo(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Mitch Hedburg.WMV
deleting Comedy Central Presents/Comedy Central Presents - Mitch Hedburg uncut.avi
deleting Comedy Central Presents/Comedy Central Presents - Mitch Fatel.avi
deleting Comedy Central Presents/Comedy Central Presents - Mike Britt.wmv
deleting Comedy Central Presents/Comedy Central Presents - Mike Birbiglia.avi
deleting Comedy Central Presents/Comedy Central Presents - Mike Birbiglia(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Megan Mooney.avi
deleting Comedy Central Presents/Comedy Central Presents - Mark Curry.avi
deleting Comedy Central Presents/Comedy Central Presents - Mario Cantone.avi
deleting Comedy Central Presents/Comedy Central Presents - Maria Bamford.wmv
deleting Comedy Central Presents/Comedy Central Presents - Maria Bamford(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Margaret Smith.avi
deleting Comedy Central Presents/Comedy Central Presents - Marc Maron.avi
deleting Comedy Central Presents/Comedy Central Presents - Marc Maron(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Lynne Koplitz.avi
deleting Comedy Central Presents/Comedy Central Presents - Louis Ramey.avi
deleting Comedy Central Presents/Comedy Central Presents - Loni Love.avi
deleting Comedy Central Presents/Comedy Central Presents - Lizz Winstead.avi
deleting Comedy Central Presents/Comedy Central Presents - Lisa Landry.avi
deleting Comedy Central Presents/Comedy Central Presents - Lenny Clarke.avi
deleting Comedy Central Presents/Comedy Central Presents - Laura Kightlinger.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Grooms.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Dunnigan.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Cease.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Nealon.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Hart.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Brennan.avi
deleting Comedy Central Presents/Comedy Central Presents - Keith Robinson.avi
deleting Comedy Central Presents/Comedy Central Presents - Kathleen Madigan.avi
deleting Comedy Central Presents/Comedy Central Presents - Judy Gold.avi
deleting Comedy Central Presents/Comedy Central Presents - Josh Sneed.avi
deleting Comedy Central Presents/Comedy Central Presents - Jon Reep.avi
deleting Comedy Central Presents/Comedy Central Presents - John Heffron.divx
deleting Comedy Central Presents/Comedy Central Presents - John Heffron(2).avi
deleting Comedy Central Presents/Comedy Central Presents - John Caparulo.avi
deleting Comedy Central Presents/Comedy Central Presents - Joey Kola.avi
deleting Comedy Central Presents/Comedy Central Presents - Joe Matarese.avi
deleting Comedy Central Presents/Comedy Central Presents - Jo Koy.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Shubert.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Pardo.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Dore.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Carr.avi
deleting Comedy Central Presents/Comedy Central Presents - Jim Gaffigan.asf
deleting Comedy Central Presents/Comedy Central Presents - Jim David.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeremy Hotz.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeff Stilson.mpg
deleting Comedy Central Presents/Comedy Central Presents - Jeff Garcia.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeff Dunham.mpg
deleting Comedy Central Presents/Comedy Central Presents - Jeff Cesario.avi
deleting Comedy Central Presents/Comedy Central Presents - Jamie Lissow.avi
deleting Comedy Central Presents/Comedy Central Presents - Jake Johannsen.avi
deleting Comedy Central Presents/Comedy Central Presents - Jackie Kashian.avi
deleting Comedy Central Presents/Comedy Central Presents - Jack Coen.avi
deleting Comedy Central Presents/Comedy Central Presents - Ian Bagg.avi
deleting Comedy Central Presents/Comedy Central Presents - Hugh Fink.avi
deleting Comedy Central Presents/Comedy Central Presents - Howard Kremer.avi
deleting Comedy Central Presents/Comedy Central Presents - Henry Phillips.avi
deleting Comedy Central Presents/Comedy Central Presents - Harland Williams.avi
deleting Comedy Central Presents/Comedy Central Presents - Hard'N'Phirm.avi
deleting Comedy Central Presents/Comedy Central Presents - Gregg Rogell.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Proops.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Giraldo.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Giraldo(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Fitzsimmons[2].avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Fitzsimmons.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Behrendt.avi
deleting Comedy Central Presents/Comedy Central Presents - Godfrey.avi
deleting Comedy Central Presents/Comedy Central Presents - Gilbert Gottfried.avi
deleting Comedy Central Presents/Comedy Central Presents - Gene Pompa.avi
deleting Comedy Central Presents/Comedy Central Presents - Gary Valentine.avi
deleting Comedy Central Presents/Comedy Central Presents - Freddy Soto.avi
deleting Comedy Central Presents/Comedy Central Presents - Frank Caliendo.wmv
deleting Comedy Central Presents/Comedy Central Presents - Finesse Mitchell.avi
deleting Comedy Central Presents/Comedy Central Presents - Eugene Mirman.avi
deleting Comedy Central Presents/Comedy Central Presents - Elvira Kurt.avi
deleting Comedy Central Presents/Comedy Central Presents - Eddie Gossling.avi
deleting Comedy Central Presents/Comedy Central Presents - Eddie Brill.avi
deleting Comedy Central Presents/Comedy Central Presents - Earthquake.avi
deleting Comedy Central Presents/Comedy Central Presents - Dwayne Perkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Dwayne Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Drew Fraser.avi
deleting Comedy Central Presents/Comedy Central Presents - Dov Davidoff.avi
deleting Comedy Central Presents/Comedy Central Presents - Doug Stanhope.avi
deleting Comedy Central Presents/Comedy Central Presents - Doug Benson.avi
deleting Comedy Central Presents/Comedy Central Presents - Don DC Curry.mpg
deleting Comedy Central Presents/Comedy Central Presents - Dom Irrera.avi
deleting Comedy Central Presents/Comedy Central Presents - Deon Cole.avi
deleting Comedy Central Presents/Comedy Central Presents - DeRay.avi
deleting Comedy Central Presents/Comedy Central Presents - David Feldman.avi
deleting Comedy Central Presents/Comedy Central Presents - Dat Phan.avi
deleting Comedy Central Presents/Comedy Central Presents - Darrell Hammond.avi
deleting Comedy Central Presents/Comedy Central Presents - Daniel Tosh.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Naturman.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Mintz.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Cummins.avi
deleting Comedy Central Presents/Comedy Central Presents - D.C. Benny.divx
deleting Comedy Central Presents/Comedy Central Presents - Craig Shoemaker.avi
deleting Comedy Central Presents/Comedy Central Presents - Corey Kahaney.avi
deleting Comedy Central Presents/Comedy Central Presents - Corey Holcomb.avi
deleting Comedy Central Presents/Comedy Central Presents - Clinton Jackson.avi
deleting Comedy Central Presents/Comedy Central Presents - Christian Finnegan.avi
deleting Comedy Central Presents/Comedy Central Presents - Chelsea Handler.avi
deleting Comedy Central Presents/Comedy Central Presents - Carol Leifer.avi
deleting Comedy Central Presents/Comedy Central Presents - Carlos Alazraqui.avi
deleting Comedy Central Presents/Comedy Central Presents - Bruce Bruce.avi
deleting Comedy Central Presents/Comedy Central Presents - Brian Regan.mpg
deleting Comedy Central Presents/Comedy Central Presents - Bob Oschack.avi
deleting Comedy Central Presents/Comedy Central Presents - Bob Marley.avi
deleting Comedy Central Presents/Comedy Central Presents - Bill Santiago.avi
deleting Comedy Central Presents/Comedy Central Presents - Bill Burr.avi
deleting Comedy Central Presents/Comedy Central Presents - Bil Dwyer.avi
deleting Comedy Central Presents/Comedy Central Presents - Big Jay Oakerson.avi
deleting Comedy Central Presents/Comedy Central Presents - Ben Bailey.avi
deleting Comedy Central Presents/Comedy Central Presents - Arnez J..avi
deleting Comedy Central Presents/Comedy Central Presents - Arj Barker[2].avi
deleting Comedy Central Presents/Comedy Central Presents - Arj Barker.avi
deleting Comedy Central Presents/Comedy Central Presents - Aries Spears.avi
deleting Comedy Central Presents/Comedy Central Presents - Ardal O'Hanlon.avi
deleting Comedy Central Presents/Comedy Central Presents - Andy Kindler.avi
deleting Comedy Central Presents/Comedy Central Presents - Andrew Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Alonzo Bodden.avi
deleting Comedy Central Presents/Comedy Central Presents - Al Madrigal.avi
deleting Comedy Central Presents/Comedy Central Presents - Adele Givens.wmv
deleting Comedy Central Presents/Comedy Central Presents - Adam Ferrara.avi
deleting Comedy Central Presents/.DS_Store
deleting Comedy Central Presents/
deleting Chubby Brown/Chubby Brown - Bad Taste [2003].avi
deleting Chubby Brown/
deleting Chris Rock/Chris Rock Big Ass Jokes.avi
deleting Chris Rock/Chris Rock - Never Scared [2004].avi
deleting Chris Rock/Chris Rock - Kill the messenger.avi
deleting Chris Rock/Chris Rock - Bring the pain 1996.divx
deleting Chris Rock/Chris Rock - Bigger And Blacker (2002).avi
deleting Chris Rock/
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/Extras/fico-cmtbofif-deleted-scenes.avi
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/Extras/
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/fico-cmtbofif.nfo
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/fico-cmtbofif.avi
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/
deleting Carlos Mencia/carlos mencia3.avi
deleting Carlos Mencia/Comedy Central Presents - Carlos Mencia.mpg
deleting Carlos Mencia/Carlos Mencia2.divx
deleting Carlos Mencia/Carlos Mencia.divx
deleting Carlos Mencia/
deleting Brian Regan/Brian Regan - Live Mp3/playlist.m3u
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - You Too Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Whale Noises.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Stupid in School.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Spider Webs Bees.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Seatbelts.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Peanut Butter Jelly.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Monster Truck Drivers.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Lousy in Little League.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Log Trucks.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Kids Party Games.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Horns Windshields.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Hooked on Phonix.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Health Club Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Flipper Gentle Ben.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Fishing on TV.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Evel Knievel.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Elevators Faces.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Donut Lady.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Dog Barking.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Crank Calls.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Blasting Zone.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Big Family Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Belly Buttons.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Animals.mp3
deleting Brian Regan/Brian Regan - Live Mp3/
deleting Brian Regan/Brian Regan - Standing Up.avi
deleting Brian Regan/Brian Regan - I Walked on the Moon.avi
deleting Brian Regan/Brian Regan - Epitome Of Hyperbole.avi
deleting Brian Regan/
deleting Brendon Burns/brendon.burns.so.i.suppose.this.is.offensive.now.2008.ws.dvdrip.xvid-pvr.nfo
deleting Brendon Burns/Brendon Burns - So I Suppose THIS Is Offensive Now.avi
deleting Brendon Burns/
deleting Bonnie McFarlane/Bonnie McFarlane - One Night Stand.avi
deleting Bonnie McFarlane/
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-51/rsync/rsync.c(244) [sender=2.6.9]

THANKS MAN THAT TOOK A SECOND BEFORE I REALISED WHAT HAPPENED NO WARNING ALL THAT DATA GONE...
@casmarc

This comment has been minimized.

Sign in to view
Copy link

casmarc commented Dec 16, 2016

That why you dont just paste anything you read on the internet in your terminal. LOL
@pando85

This comment has been minimized.

Sign in to view
Copy link

pando85 commented Dec 24, 2016

To allow arcfour cipher:

sshd config

Add at the end of /etc/sshd_config file:

# enable all ciphers!
# obtained with ssh -Q cipher localhost | paste -d , -s
Ciphers 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,arcfour128,arcfour256,aes128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
@claytondukes

This comment has been minimized.

Sign in to view
Copy link

claytondukes commented Jan 4, 2017

I'm only getting marginally better transfers. Would anyone happen to know why?

here's what I have:

# Rsync options:
src='user@host:/dir/'
dest='/dir/'
excludefile='/root/.ignore-sync'
deleteOld="--delete-before --delete-excluded --max-delete=50"
options=(-aHAXxv --progress --partial --append --stats --min-size=1 --prune-empty-dirs --numeric-ids ${deleteOld} --exclude-from=${excludefile} ${limit} -e 'ssh -T -c arcfour -o Compression=no -x')

rsync "${options[@]}" ${src} ${dest}

Prior to using your settings, I used:

syncCmd="rsync -rltvhm --progress --partial --append --stats --min-size=1 --prune-empty-dirs ${deleteOld} --exclude-from=${excludefile} ${limit} ${src} ${dest}"

Before using yours I got around 4mbps
now I get around 6mbps.
As you can see below, I should have plenty of bandwidth.

Link speedtests
Host (Upload Speed used here):

# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Google Fiber (136.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Google Fiber: 18.029 ms
Testing download speed................................................................................
Download: 477.76 Mbit/s
Testing upload speed....................................................................................................
Upload: 173.21 Mbit/s

Client (Download Speed used here):

# speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from XFINITY (73.x.x.x)...
Selecting best server based on latency...
Hosted by CenturyLink, Inc: 47.933 ms
Testing download speed........................................
Download: 85.72 Mbit/s
Testing upload speed..................................................
Upload: 11.77 Mbit/s
@thesandbender

This comment has been minimized.

Sign in to view
Copy link

thesandbender commented Mar 21, 2017 •
edited

arcfour (RC4) is disabled for a reason. There are known, working exploits against it that allow RC4 encrypted data to be deciphered. If you have a AMD or Intel CPU that was made after 2010-2011 it probably supports AES-NI and you're better off using that as @aflyhorse suggested. You should never intentionally weaken a system's security, even if "it's just for a little bit".
@henryx

This comment has been minimized.

Sign in to view
Copy link

henryx commented May 25, 2017

Use -P instead --progress, because this is equivalent for --progress --partial
@toloveru

This comment has been minimized.

Sign in to view
Copy link

toloveru commented Jun 17, 2017 •
edited

Thanks a lot! Backup speeds to my local server improved significantly! Now it hits 60 MB/s easily 😀
@dukelambert

This comment has been minimized.

Sign in to view
Copy link

dukelambert commented Jun 28, 2017

Really thanks for this post, the transfer rate is improved up to 90 MB/s . I'd like to know how to place '--exclude /backup/somedir' in the string because it doesn't work. Thanks in advance.
@ktavabi

This comment has been minimized.

Sign in to view
Copy link

ktavabi commented Aug 20, 2017

Thanks for this! I hit >60 MB/S rates without the weak SSH cipher. Don't think -c arcfour is really necessary.
@denysvitali

This comment has been minimized.

Sign in to view
Copy link

denysvitali commented Nov 26, 2017

@HaniKali Same here ahah. That's why you shouldn't copy & paste random commands from the internet. I assumed --delete was used to delete remote files once they were synced. I "lost" 200+ GB of music files on my Plex server, but I have a backup, so it isn't really a big deal 🤣
@tomrunia

This comment has been minimized.

Sign in to view
Copy link

tomrunia commented Nov 29, 2017

Since -c arcfour is considered obsolete, using -c aes128-gcm@openssh.com is probably fastest.

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" [source_directory] user@hostname:[target_directory]/

See these encryption benchmark results: https://famzah.files.wordpress.com/2015/06/openssh-ciphers-performance-2015-chart.jpg
@janmd

This comment has been minimized.

Sign in to view
Copy link

janmd commented Dec 6, 2017

Hi,

currently transferring from one server to another of 1.5TB. I use this command

rsync -av --progress /home/user/public_html/* :/home/user/public_html/

I started it around 6hrs ago. Source server is having 100Mbit port speed. Destination is having 1Gbit. I issue the command from the source server.

I checked my server output bandwidth and it reports average 25Mbps for the last 4hrs

Is it still worth stopping it and using this command?

Also, I have no existing backup. Will this command remove the source? Im a bit worried because of the post by HaniKali.

This is my first time using rsync and I don't want to lose anything from source.

Can someone confirm if this will not delete my source? What happened to HaniKali? Why his files was deleted by using the command?

Thanks
@okhosting

This comment has been minimized.

Sign in to view
Copy link

okhosting commented Dec 7, 2017

Very usefull post, thanks!
@genokan

This comment has been minimized.

Sign in to view
Copy link

genokan commented Dec 9, 2017

@janmd..

Read the man page to understand what each thing is doing.

   --delete
          This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized.  You must have asked
          rsync  to  send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus
          gets a request to transfer individual files, not the files’ parent directory.  Files that are excluded from the transfer  are  also  excluded  from  being  deleted  unless  you  use  the
          --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

          Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled.  Beginning with 2.6.7, deletions will also occur when --dirs (-d) is enabled, but only for directo‐
          ries whose contents are being copied.

          This option can be dangerous if used incorrectly!  It is a very good idea to first try a run using the --dry-run option (-n) to see what files are going to be deleted.

          If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. This is to prevent temporary filesystem  failures  (such  as
          NFS errors) on the sending side from causing a massive deletion of files on the destination.  You can override this with the --ignore-errors option.

          The  --delete  option  may be combined with one of the --delete-WHEN options without conflict, as well as --delete-excluded.  However, if none of the --delete-WHEN options are specified,
          rsync will choose the --delete-during algorithm when talking to rsync 3.0.0 or newer, and the --delete-before algorithm when talking to an  older  rsync.   See  also  --delete-delay  and
@alinz

This comment has been minimized.

Sign in to view
Copy link

alinz commented Jan 20, 2018

Just want to say awesome. Thank you, I manage to upload really fast on very slow connection!
@felixlindemann

This comment has been minimized.

Sign in to view
Copy link

felixlindemann commented Feb 14, 2018 •
edited

Thank you! made my Day. Improved speed by the factor of 8.5 :-D
I had to modify it slightly to my setup qnap (new device) wdmycloud(old device) as the encryption wasn't accepted

you can find my version here
rsync -rtXx --numeric-ids --progress -e "ssh -T -c aes128-ctr -o Compression=no -x" <user>@<ip>:/<source-path>/ /<dest.path>/
https://gist.github.com/felixlindemann/dfee61a20bbd838cdb26f44cc5552404
@jjuanmera

This comment has been minimized.

Sign in to view
Copy link

jjuanmera commented Mar 22, 2018

I was looking a way to speed up the transfer between mac os and Raspberry pi. This message is after input the password:

rsync: on remote machine: -vlHogDtpAXrxe.iLsfx: unknown option
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(1333) [server=2.6.9]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

Any ideas? Thanks
@tufahu

This comment has been minimized.

Sign in to view
Copy link

tufahu commented May 3, 2018

thank you! very good hints!
@win4me

This comment has been minimized.

Sign in to view
Copy link

win4me commented May 10, 2018 •
edited

Comparing the methods in dust-free lab over the 10 Gbit "jumbo frame" network link:

-   13.6 MB/s  rsync -az 
-   16.7 MB/s  scp -Cr
-   44.8 MB/s  rsync -a 
-   59.8 MB/s  sftp
-   61.2 MB/s  scp -r 
-   61.4 MB/s  sftp  -R 128 -B 65536
-   62.4 MB/s  rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"
-  143.5 MB/s  scp -r -c arcfour
-  144.2 MB/s  sftp -oCiphers=arcfour

Read details of the benchmark test, commands, and output in my post: http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html
@teusbenschop

This comment has been minimized.

Sign in to view
Copy link

teusbenschop commented Jun 6, 2018

Thank you, the speedup makes a good difference!
@danielmotaleite

This comment has been minimized.

Sign in to view
Copy link

danielmotaleite commented Jul 12, 2018

If you can, please put the --delete descripiton in bold and with a bigger warming, so people like HaniKali so not remove delete valid content.

There are always people that just copy&paste, with bigger warnings is totally their fault!
@Cho1409

This comment has been minimized.

Sign in to view
Copy link

Cho1409 commented Jul 13, 2018

Witchcraft! Thank you!
@areeb111

This comment has been minimized.

Sign in to view
Copy link

areeb111 commented Aug 6, 2018

Thank you, I got 110MB/s same as your command but only with rsync -av and ssh attributes
@seandex

This comment has been minimized.

Sign in to view
Copy link

seandex commented Aug 22, 2018

is arcfour even possible in 2018?

cat /proc/cpuinfo | grep aes
if you have it then use AES ciphers.
moreover, if both-end have it, you can achieve it up to 900Mbps.
@harish2704

This comment has been minimized.

Sign in to view
Copy link

harish2704 commented Aug 23, 2018

Thanks for sharing this information.

I tried this and this is what I understood.

  • Network bandwidth utilization doesn't represent amount of data transferred.
  • When I was using this script, I got 90Mb/s network usage and 11MB/s data transfer.
  • When I removed -o compression=no from ssh , data transfer increased to 20MB/s
  • When I enabled z option in rsync datatransfer rate increased to 29MB/s ( and network utilization was 60Mb/s )

Lesson learned:

  • This script may be helpful to when we have un-compressible data and we have Gigabit NIC.
  • but when network becomes bottleneck and/or your data is compressible , this method may not work well.
@scottinan

This comment has been minimized.

Sign in to view
Copy link

scottinan commented Sep 13, 2018

Thanks! From an SSD mdadm array on ext4 to a ZFS raid z2 with 2TBs spinning disks I'm getting maxing out at 1GB lol gonna have to set up a bonded interface! 👍
@sjuxax

This comment has been minimized.

Sign in to view
Copy link

sjuxax commented Sep 22, 2018

is arcfour even possible in 2018?

cat /proc/cpuinfo | grep aes
if you have it then use AES ciphers.
moreover, if both-end have it, you can achieve it up to 900Mbps.

arcfour is not compiled in to my OpenSSH daemon. chacha20-poly1305@openssh.com is the modern replacement. I get ~100MB/s with that v. capping out around 40MB/s with default ciphers (AES).
@88plug

This comment has been minimized.

Sign in to view
Copy link

88plug commented Oct 1, 2018 •
edited

rsync -av --progress -e "ssh -T -c aes128-ctr -o Compression=no -x" source/ user@x.x.x.x:/destination

The key is leaving off the compression (-z) and using aes128-ctr on Ubuntu 18 (latest cipher support)
@MBetters

This comment has been minimized.

Sign in to view
Copy link

MBetters commented Dec 4, 2018 •
edited

I'm using this script. It gets rid of the extended attributes ("-X" flag) if you're running this on Windows. It also uses the aes128-gcm cipher in the likely case that your openssh installation doesn't include arcfour.

#!/bin/bash
# Fast rsync command

# Set the RSYNC_ARGS.
UNAME="$(uname -s)"
case "${UNAME}" in 
    Linux* | Darwin*)
        RSYNC_ARGS="-aHAXxv --numeric-ids --delete --progress -e"
    ;;
    # Windows filesystems do not support extended attributes (the "-X" option)
    CYGWIN* | MINGW*)
        RSYNC_ARGS="-aHAxv --numeric-ids --delete --progress -e"
    ;;
    *)
        echo "ERROR: Running on unknown system! Exiting!"
        return 1
    ;;
esac

# Set the SSH_ARGS
SSH_ARGS="-T -o -c aes128-gcm@openssh.com Compression=no -x"

# Get the rest of the args from the caller
USER=$1
SOURCE=$2
SOURCE_DIR=$3
DEST_DIR=$4

rsync $RSYNC_ARGS "ssh $SSH_ARGS" $USER@$SOURCE:$SOURCE_DIR $DEST_DIR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment