Skip to content

Instantly share code, notes, and snippets.

@KartikTalwar

KartikTalwar/Documentation.md

Last active Mar 4, 2021
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Rsync over SSH - (40MB/s over 1GB NICs)
Raw
Documentation.md

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

  • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
  • H: preserves hard-links
  • A: preserves ACLs
  • X: preserves extended attributes
  • x: don't cross file-system boundaries
  • v: increase verbosity
  • --numeric-ds: don't map uid/gid values by user/group name
  • --delete: delete extraneous files from dest dirs (differential clean-up during sync)
  • --progress: show progress during transfer

ssh

  • T: turn off pseudo-tty to decrease cpu load on destination.
  • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.
  • o Compression=no: Turn off SSH compression.
  • x: turn off X forwarding if it is on by default.

Original

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>

Flip

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]
Raw
SSHrsync.sh
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>
@hollodk

This comment has been minimized.

Sign in to view
Copy link

@hollodk hollodk commented Nov 20, 2013

thanks.. you just saved me a few hours.
@reox

This comment has been minimized.

Sign in to view
Copy link

@reox reox commented Dec 25, 2013

nice, got up to 64MB/s over Gbit Network.
faster is only the direct netcat approach (about 80MB/s):
on the sending side: tar -c /path/to/data | nc -q 10 12345
on the receiving side: nc -w 10 -l -p 12345 | tar -x
but thats obviously unencrypted.
@cleytonferrari

This comment has been minimized.

Sign in to view
Copy link

@cleytonferrari cleytonferrari commented Mar 19, 2014

Muito Obrigado!
@TheMengzor

This comment has been minimized.

Sign in to view
Copy link

@TheMengzor TheMengzor commented Jul 6, 2014

MAAAAAAAAD!
Thank you so much!
@reox

This comment has been minimized.

Sign in to view
Copy link

@reox reox commented Oct 12, 2014

by the way: i could observe 110MB/s with intel NICs and SSD to SSD transfer. So the real bottleneck is only your harddisk
@gozoinks

This comment has been minimized.

Sign in to view
Copy link

@gozoinks gozoinks commented Jan 25, 2015

Hard disk performance is not remotely the only bottleneck. Encryption and compression algorithms have wildly varying performance as well. All else being equal, SSD or no, using faster/lighter encryption and faster or no compression can improve payload throughput considerably. And it has as much to do with the material being transferred as anything else, particularly if the source is difficult to compress or encrypt. These things can't be trivialized, and careful attention to SSH options can be very important for performance.

Thanks to the OP. T c o and x are valuable options to use for bulk transfers.
@denji

This comment has been minimized.

Sign in to view
Copy link

@denji denji commented Mar 2, 2015 

ciphers arcfour256,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
@bubenkoff

This comment has been minimized.

Sign in to view
Copy link

@bubenkoff bubenkoff commented Mar 8, 2015

thanks a lot for this gist!
@priyadarshan

This comment has been minimized.

Sign in to view
Copy link

@priyadarshan priyadarshan commented Mar 20, 2015

If you use the rsync:// protocol scheme (i.e. when you connect to a rsyncd daemon) then no encryption will be used (although password authentication is done using a MD4-based challenge-response system and is probably still reasonably secure). See: http://unix.stackexchange.com/questions/191249/understanding-why-such-a-discrepancy-in-network-transfer
@rickw

This comment has been minimized.

Sign in to view
Copy link

@rickw rickw commented May 28, 2015

Thanks, MacPro to Ubuntu Box is going at 100+MB/s
@KevinMKorb

This comment has been minimized.

Sign in to view
Copy link

@KevinMKorb KevinMKorb commented Jul 22, 2015

I understand the desire for using a weak cipher on an internal network but at least use arcfour128. It has essentially the same performance as arcfour (both are 128-bit) but the random session key will actually be random rather than guessable.
@mmitchell816

This comment has been minimized.

Sign in to view
Copy link

@mmitchell816 mmitchell816 commented Aug 5, 2015

Thanks for this. I was only using vaHz flags and was getting about 15MB between two server each with 8 HDDs, now I'm hitting 100MB+. This just saved me days off my transfer. And thats without changing sshd_config on the remote server. Not that it would help at this point since I'm nearly maxing out the Gig link.
@gpmcadamold

This comment has been minimized.

Sign in to view
Copy link

@gpmcadamold gpmcadamold commented Aug 12, 2015

@KartikTalwar This is a life-saver, thanks. However, there's a typo in the description:

--numeric-ds

Should be:

--numeric-ids
@2manydjs

This comment has been minimized.

Sign in to view
Copy link

@2manydjs 2manydjs commented Aug 26, 2015

Unfortunately I did not get the same improvement between my OS X box and Synology NAS. In specific -e "ssh -c arcfour -o Compression=no" will make it slow down from 17MB/s to 13MB/s (on Gbit).
@kedare

This comment has been minimized.

Sign in to view
Copy link

@kedare kedare commented Aug 30, 2015

Got 80MB/s between OS X and a Proliant Microserver n40l :)
@kroko23

This comment has been minimized.

Sign in to view
Copy link

@kroko23 kroko23 commented Dec 13, 2015

I think is faster to mount remote server with nfs and do rsync locally.
@AntonMay

This comment has been minimized.

Sign in to view
Copy link

@AntonMay AntonMay commented Jan 5, 2016

41,200,648,192 100% 94.87MB/s 0:06:54

Awesome!!!!
@minhaz1

This comment has been minimized.

Sign in to view
Copy link

@minhaz1 minhaz1 commented Jan 10, 2016

I was able to get around 100 MB/s running rsync in daemon mode in parallel w/ 4 processes, transferring around 1.5 million 800kb files. I'm confident that the 1 Gbps nic was the bottleneck since we have really fast Virident SSD's. With 10-gig nic I probably could get several hundred MB/s.
@aflyhorse

This comment has been minimized.

Sign in to view
Copy link

@aflyhorse aflyhorse commented Feb 25, 2016

Since OpenSSH 6.7, the "arcfour" cipher was disabled by default. For example, the sshd ship with Debian Jessie:

ssh -c arcfour sso
Unable to negotiate with x.x.x.x: no matching cipher found. Their offer: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com

For newer hardware (both sides have CPU with AES-NI), I recommend "aes128-gcm@openssh.com" as an alternative.
@kosir

This comment has been minimized.

Sign in to view
Copy link

@kosir kosir commented Mar 10, 2016

Kudos! Works perfectly, super fast! 👍
@kosir

This comment has been minimized.

Sign in to view
Copy link

@kosir kosir commented Mar 10, 2016

How to exclude (for example) hidden directories? Anything I tried doesn't work.
@Fossil01

This comment has been minimized.

Sign in to view
Copy link

@Fossil01 Fossil01 commented May 18, 2016

Sped my transfer up from ~10 MB/s to 30 MB/s from SATA RAID-1 to SATA RAID-1. Will try later with SSD to SSD.
@mautematico

This comment has been minimized.

Sign in to view
Copy link

@mautematico mautematico commented Jun 10, 2016

sent 1,078 bytes received 167,802 bytes 48,251.43 bytes/sec, over Internet.

My server does not support arcfour, so I removed that option. :)
@stefancrain

This comment has been minimized.

Sign in to view
Copy link

@stefancrain stefancrain commented Jul 19, 2016 •
edited

Great notes @KartikTalwar! I skip compression on filetypes that I know are encrypted and not worth the CPU time to compress. As a result transfers move much fast on my machines.

export RSYNC_SKIP_COMPRESS=3fr/3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/arw/asf/avi/bz2/cab/cr2/crypt[5678]/dat/dcr/deb/dmg/drc/ear/erf/flac/flv/gif/gpg/gz/iiq/iso/jar/jp2/jpeg/jpg/k25/kdc/lz/lzma/lzo/m4[apv]/mef/mkv/mos/mov/mp[34]/mpeg/mp[gv]/msi/nef/oga/ogg/ogv/opus/orf/pef/png/qt/rar/rpm/rw2/rzip/s7z/sfx/sr2/srf/svgz/t[gb]z/tlz/txz/vob/wim/wma/wmv/xz/zip
rsync --skip-compress=$RSYNC_SKIP_COMPRESS .....

Cheers!
@RedShift1

This comment has been minimized.

Sign in to view
Copy link

@RedShift1 RedShift1 commented Jul 30, 2016

Nice
@gabrielfsousa

This comment has been minimized.

Sign in to view
Copy link

@gabrielfsousa gabrielfsousa commented Aug 17, 2016

thanks a lot
@rorymcdaniel

This comment has been minimized.

Sign in to view
Copy link

@rorymcdaniel rorymcdaniel commented Sep 5, 2016

Wow, thanks for sharing these, I can't believe how much faster it is.
@therealmarv

This comment has been minimized.

Sign in to view
Copy link

@therealmarv therealmarv commented Sep 16, 2016

for a Mac to Linux transfer it's useful to use other options. Arcfour is not available for most new machines anymore and UTF-8 on OS X is different than UTF-8 on Linux (important if you have Umlauts like Germans, Samba/NFS will fail otherwise). My command if both (Mac & Linux) machines support AES on their processors and you want to transfer from Mac to Linux:

rsync -rltv --progress --human-readable --delete --iconv=utf-8-mac,utf-8 -e 'ssh -T -c aes128-gcm@openssh.com -o Compression=no -x' <local_mac_source> <remote_linux_dest>

reverse the iconv option if you want to transfer from Linux to Mac.
@klein0r

This comment has been minimized.

Sign in to view
Copy link

@klein0r klein0r commented Oct 26, 2016

Well done! Thanks a lot :) Speeds up a copy process from Synology NAS to NAS a lot!
@HaniKali

This comment has been minimized.

Sign in to view
Copy link

@HaniKali HaniKali commented Dec 15, 2016

Mac-mini:~ hani2$ rsync -a -v --remove-source-files --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" /Users/hani2/Desktop/3TB/Stand\ Up\ Comedy/ /Users/hani2/Desktop/DOCO+STANDUP/Stand\ Up\ Comedy/
building file list ...
188 files to consider
deleting def comedy jams/def.comedy.jam.all.stars.vol.09.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.06.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.05.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.04.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.03.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.02.avi
deleting def comedy jams/def.comedy.jam.all.stars.vol.01.avi
deleting def comedy jams/NFO.nfo
deleting def comedy jams/Def Comedy Jam All. Stars vol.08.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.12.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.11.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.10.avi
deleting def comedy jams/Def Comedy Jam All Stars vol.07.avi
deleting def comedy jams/
deleting Louis CK/Louis C.K. Interview.flv
deleting Louis CK/Louis C.K. 1995.flv
deleting Louis CK/Louis C.K. - Shameless.avi
deleting Louis CK/
deleting Lisa Lampanelli/Lisa Lampanelli - Long Live the Queen.avi
deleting Lisa Lampanelli/
deleting Lewis Black/MP3/The White Album/Lewis Black - 12 - Other Idiots from Arkansas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 11 - Education in Arkansas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 10 - The Impeachment.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 09 - Bill Clinton and Oral Sex.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 08 - Heaven's Gate.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 07 - IHOP (The International House of Pancakes).mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 06 - TV Pilot.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 05 - Fuck, New York, Los Angeles.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 04 - Miami and Las Vegas.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 03 - The Ozone, Sunblock, The Flu, and Nyquil.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 02 - The Fall, Hurricanes, and Weathermen.mp3
deleting Lewis Black/MP3/The White Album/Lewis Black - 01 - Wisconsin.mp3
deleting Lewis Black/MP3/The White Album/
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 13 - A Sense of Humor.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 12 - The War in Iraq.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 11 - Bringing Democracy to Iraq.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 10 - Homeland Security.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 09 - Who's Fucking Who.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 08 - Greed.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 07 - Small Pox.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 06 - Health Clubs.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 05 - Ireland & Health.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 04 - International Travel.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 03 - The Coldest Winter Ever.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 02 - The Settling of Minnesota.mp3
deleting Lewis Black/MP3/Rules of Enragement/Lewis Black - 01 - Minnesota Winters.mp3
deleting Lewis Black/MP3/Rules of Enragement/
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 10 - What I've Learned.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 09 - TV News & Jerry Falwell.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 08 - Airport Security.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 07 - Tax Rebates and Common Sense.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 06 - The End of the Universe.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 05 - Halftime At the Super Bowl 200.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 04 - Coke and Pepsi and the Super B.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 03 - Environmental Terrorism or Glo.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 02 - After September 11th.mp3
deleting Lewis Black/MP3/End of the Universe/Lewis Black - 01 - Atlanta.mp3
deleting Lewis Black/MP3/End of the Universe/
deleting Lewis Black/MP3/
deleting Lewis Black/Lewis Black on Brodway.avi
deleting Lewis Black/Lewis Black - Taxed Beyond Belief.avi
deleting Lewis Black/Lewis Black - Red White and Screwed.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black 2002.avi
deleting Lewis Black/Comedy Central Presents - Lewis Black 2000.avi
deleting Lewis Black/
deleting Lee Evans/Big Live cd2 [2008].avi
deleting Lee Evans/Big Live cd1 [2008].avi
deleting Lee Evans/
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 5).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 4).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 3).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 2).avi
deleting Laughing Out Loud/Laughing Out Loud - Standup Comedy (vol. 1).avi
deleting Laughing Out Loud/
deleting Latin Divas of Comedy/Latin Divas.divx
deleting Latin Divas of Comedy/
deleting Laffapalooza/Laffapalooza 3 [2007].avi
deleting Laffapalooza/Laffapalooza 2 [2007].avi
deleting Laffapalooza/Laffapalooza 1 [2006].avi
deleting Laffapalooza/
deleting Kyle Cease/Kyle Cease - Weirder Blacker Dimpler [2007].avi
deleting Kyle Cease/
deleting Kings of Comedy/The Original Kings Of Comedy.avi
deleting Kings of Comedy/
deleting Kevin James/Kevin James, sweat the small stuff.mpg
deleting Kevin James/
deleting Katt Williams/Katt Williams - Live [2006].avi
deleting Katt Williams/Katt Williams - Its Pimpin Pimpin [2008].avi
deleting Katt Williams/
deleting Just for Laughs/Just for Laughs 2004 Montreal Comedy Festival.avi
deleting Just for Laughs/
deleting Joey Coco Diaz/Joey Coco Diaz Socially Unacceptable 2016 720p SESO WEB.mp4
deleting Joey Coco Diaz/
deleting Jo Koy/Dont Make Him Angry [2009].avi
deleting Jo Koy/
deleting Jimmy Carr/Jimmy_Carr_--Comedy_Central_stand_up--_Part_2.avi
deleting Jimmy Carr/Jimmy Carr - Stand Up.avi
deleting Jimmy Carr/Jimmy Carr - Live.avi
deleting Jimmy Carr/Jimmy Carr - Live at the Apollo (2).avi
deleting Jimmy Carr/Jimmy Carr - Live at the Apollo (1).avi
deleting Jimmy Carr/Jimmy Carr - In Concert.avi
deleting Jimmy Carr/Jimmy Carr - Comedian .avi
deleting Jimmy Carr/
deleting Jim Gaffigan/Jim Gaffigan - King Baby.avi
deleting Jim Gaffigan/Jim Gaffigan - Beyond The Pale.avi
deleting Jim Gaffigan/
deleting Jim Carrey/Jim Carrey - Unnatural Act (1991).avi
deleting Jim Carrey/Jim Carrey - Standup Routine - Comedy Store (1996).mpg
deleting Jim Carrey/
deleting Jim Breuer/Jim Breuer - Hardcore.avi
deleting Jim Breuer/
deleting Jerry Seinfeld/Stand up/tonight show with micheal richards.avi
deleting Jerry Seinfeld/Stand up/tonight show appearance 1990.avi
deleting Jerry Seinfeld/Stand up/stand up 1981.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 4.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 3.avi
deleting Jerry Seinfeld/Stand up/exclusive stand-up 2.avi
deleting Jerry Seinfeld/Stand up/exclusive stand up 1.avi
deleting Jerry Seinfeld/Stand up/Saturday Night Live Hosted by Jerry Seinfeld.avi
deleting Jerry Seinfeld/Stand up/
deleting Jerry Seinfeld/Seinfeld Reunion Special.avi
deleting Jerry Seinfeld/Jerry Seinfeld - The Comedian.avi
deleting Jerry Seinfeld/Jerry Seinfeld - Stand-Up Confidential 1987.avi
deleting Jerry Seinfeld/Jerry Seinfeld - Not That There's Anything Wrong With That (Bloopers From S2 DVD).avi
deleting Jerry Seinfeld/Jerry Seinfeld - I`m telling it for the last time.avi
deleting Jerry Seinfeld/
deleting Jeff Dunham/Jeff Dunhams - Very Special Christmas Special [2008].avi
deleting Jeff Dunham/Jeff Dunham - Spark of Insanity [2007].avi
deleting Jeff Dunham/Jeff Dunham - Arguing with Myself [2006].avi
deleting Jeff Dunham/
deleting Jason Byrne/Jason Byrne - The Byrne Identity [2008].avi
deleting Jason Byrne/
deleting Heckler/aaf-heckler.nfo
deleting Heckler/aaf-heckler.avi
deleting Heckler/
deleting George Carlin/George Carlin - The Mark Twain prize.avi
deleting George Carlin/George Carlin - Its bad for ya [2008].avi
deleting George Carlin/George Carlin - Inside The Actor's Studio [2004].avi
deleting George Carlin/George Carlin - George's Best Stuff [1996].avi
deleting George Carlin/George Carlin - 12 - Complaints and Grievances [2001].avi
deleting George Carlin/George Carlin - 11 - You Are All Diseased [1999].avi
deleting George Carlin/George Carlin - 10 - 40 Years Of Comedy [1997].avi
deleting George Carlin/George Carlin - 09 - Back In Town [1996].avi
deleting George Carlin/George Carlin - 08 - Jammin' In New York [1992].avi
deleting George Carlin/George Carlin - 07 - Doin' It Again [1990].avi
deleting George Carlin/George Carlin - 06 - What the Fuck am I Doing in New Jersey [1988].avi
deleting George Carlin/George Carlin - 05 - Playing With Your Head [1986].avi
deleting George Carlin/George Carlin - 04 - Carlin On Campus [1984].avi
deleting George Carlin/George Carlin - 02 - Again [1978].avi
deleting George Carlin/George Carlin - 01 - On Location At Usc [1977].avi
deleting George Carlin/
deleting Gabriel Iglesias/Gabriel Iglesias - Hot and Flufy.divx
deleting Gabriel Iglesias/Comedy Central Presents - Gabriel Iglesias.avi
deleting Gabriel Iglesias/
deleting Frankie Boyle/Frankie Boyle - Live [2008].avi
deleting Frankie Boyle/
deleting Frank Skinner/frank.skinner.standup.live.from.birminghams.nia.dvdrip.xvid-nosegment.nfo
deleting Frank Skinner/frank skinner - live from birminghams nia.avi
deleting Frank Skinner/
deleting Frank Caliendo/Frank Caliendo-Giggles.avi
deleting Frank Caliendo/
deleting Flight Of The Conchords/Flight Of The Conchords - HBO One Night Stand.avi
deleting Flight Of The Conchords/
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (7).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (6).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (5).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (4).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (3).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (2).avi
deleting Ellen DeGeneres/Ellen DeGeneres - Here Now (1).avi
deleting Ellen DeGeneres/
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/18 - Eddie Murphy - Seeing Birth - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/17 - Eddie Murphy - Almost Fucked a Midget - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/16 - Eddie Murphy - Black Sambo - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/15 - Eddie Murphy - Little Richard - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/14 - Eddie Murphy - Lost in Space - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/13 - Eddie Murphy - Moses - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/12 - Eddie Murphy - Niggaz in the '70s - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/11 - Eddie Murphy - Buckwheat - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/10 - Eddie Murphy - Grandma Klump - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/09 - Eddie Murphy - Skeleton in Closet - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/08 - Eddie Murphy - Cumin' Hard - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/07 - Eddie Murphy - Hit by a Car - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/06 - Eddie Murphy - Boxers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/05 - Eddie Murphy - Ice Cream Man - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/04 - Eddie Murphy - Old Jew - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/03 - Eddie Murphy - Singers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/02 - Eddie Murphy - Drinking Fathers - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/01 - Eddie Murphy - The Barbecue - Greatest Comedy Hits.mp3
deleting Eddie Murphy/Eddie Murphy Greatest Comedy Hits/
deleting Eddie Murphy/Eddie Murphy - Raw (1987).avi
deleting Eddie Murphy/Eddie Murphy - Delirious (1983).avi
deleting Eddie Murphy/
deleting Eddie Izzard/Eddie Izzard - Definite Article.avi
deleting Eddie Izzard/
deleting Eddie Griffin/Eddie Griffin - Last Stand [2003].avi
deleting Eddie Griffin/Eddie Griffin - Freedom Of Speech [2008].avi
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (4).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (3).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (2).flv
deleting Eddie Griffin/Eddie Griffin - Dysfunktional family (1).flv
deleting Eddie Griffin/.DS_Store
deleting Eddie Griffin/
deleting Dylan Moran/Dylan Moran - Monster.avi
deleting Dylan Moran/Dylan Moran - Melbourne Comedy Gala 2006.avi
deleting Dylan Moran/Dylan Moran - Like Totally.avi
deleting Dylan Moran/
deleting Drew Hastings/fico-dhiam.sfv
deleting Drew Hastings/fico-dhiam.nfo
deleting Drew Hastings/Drew Hastings - Irked and Miffed 2008.avi
deleting Drew Hastings/
deleting Dennis Miller/Dennis Miller The Raw Feed 2003.avi
deleting Dennis Miller/
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 06 Driving home that day.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 05 Someone Tell My Mom.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 04 I want all Americans.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 03 The Asshole Olympics.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 02 Number One.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/Denis Leary - 01 The Pope is a Pimp.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 5 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 12 One thing that drives.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 11 My wife screams.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 10 The big chunk.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 09 We all know women.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 08 We couldn't care less.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 07 This Is Your Brain.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 06 Does This Bomb.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 05 She loves us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 04 Semen count.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 03 I was ready to steamroll.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 02 There was an author.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/Denis Leary - 01 Grande Vente Mocha.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 4 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 12 Three words for all the.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 11 I think its fair to say.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 10 When I walked into the.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 09 Matt Dillon Is A Giant.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 08 Nuns, Tits, Booze.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 07 We decided not to put.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 06 How I Became A.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 05 How I Learned To Act.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 04 And that was the end.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 03 My advice to men.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 02 Now I know how.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/Denis Leary - 01 Autism Shmautism.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 3 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 12 Why the fuck do old.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 11 As he drew closer.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 10 Within seconds Bobby.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 09 Bullies R Us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 08 We never went hungry.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 07 I got a job in a diner.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 06 And on and on it went.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 05 I have definite and vitall.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 04 I Had Sex With.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 03 Please Drug Your Children.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 02 Stabbing screaming.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/Denis Leary - 01 Your Kids Are Not Cute.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 2 of 5/
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 18 Me.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 17 Angelina Jolie.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 16 Rehab.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 15 NASCAR.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 14 The Double Quarter Pounder.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 13 Bill Clinton Blow Jobs.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 12 You.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 11 Britney Spears's Head.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 10 Britney Spears's Vagina.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 09 George Bush Jr..mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 08 Anna Nicole Smith.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 07 Why Everyone Hates Us.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 06 Yup-there is a real.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 05 Danica Patrick.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 04 Okay, this is the prolouge.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 03 A Second Note.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 02 Secret Hidden Message.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/Denis Leary - 01 Penguin Audio presents.mp3
deleting Denis Leary/MP3/Why We Suck/Disc 1 of 5/
deleting Denis Leary/MP3/Why We Suck/
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 10 Voices in My Head.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 09 Traditional Irish Folk Song.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 08 The Downtrodden Song.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 07 Death.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 06 Meat.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 05 Smoke.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 04 More Drugs.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 03 Rehab.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 02 Drugs.mp3
deleting Denis Leary/MP3/No Cure for Cancer/Denis Leary - 01 Asshole.mp3
deleting Denis Leary/MP3/No Cure for Cancer/
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 20 Lock 'n Load.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 19 Fuck The Pope.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 18 Life's Gonna Suck.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 17 My Kids.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 16 Insane cowboy (In Africa).mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 15 Fat Fucks.MP3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 14 Love Barge.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 13 A Reading from the Book of Apple.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 12 President Leary.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 11 Fuck The Kennedys.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 10 I'm Happy.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 09 Elvis and I.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 08 Fuck Santa.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 07 Beer.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 06 Coffee.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 05 Deaf Mute Coctail Party.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 04 Save This.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 03 Marv Marv Marv.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 02 Asshole of the Dance.mp3
deleting Denis Leary/MP3/Lock 'n Load/Denis Leary - 01 Fuck This.mp3
deleting Denis Leary/MP3/Lock 'n Load/
deleting Denis Leary/MP3/
deleting Denis Leary/Denis Leary - No Cure For Cancer.avi
deleting Denis Leary/Denis Leary - Lock 'N Load.avi
deleting Denis Leary/
deleting Demetri Martin/Demetri Martin - If I (6).avi
deleting Demetri Martin/Demetri Martin - If I (5).avi
deleting Demetri Martin/Demetri Martin - If I (4).avi
deleting Demetri Martin/Demetri Martin - If I (3).avi
deleting Demetri Martin/Demetri Martin - If I (2).avi
deleting Demetri Martin/Demetri Martin - If I (1).avi
deleting Demetri Martin/Comedy Central Presents - Demetri Martin.mpg
deleting Demetri Martin/
deleting Dave Chappelle/Dave Chappelle - Killing Them Softly [2003].avi
deleting Dave Chappelle/Dave Chappelle - For what its worth [2004].divx
deleting Dave Chappelle/Dave Chapelle in San Francisco (3).avi
deleting Dave Chappelle/Dave Chapelle in San Francisco (2).avi
deleting Dave Chappelle/Dave Chapelle in San Francisco (1).avi
deleting Dave Chappelle/.DS_Store
deleting Dave Chappelle/
deleting Dave Attell/Dave Attell - Captain Miserable.avi
deleting Dave Attell/Dave Attel - Insomniac Tour [2005].avi
deleting Dave Attell/Comedy Central Presents - Dave Attell.avi
deleting Dave Attell/
deleting Dara O Brian/Dara O Brian - Talks Funny Live In London [2007].avi
deleting Dara O Brian/Dara O Briain - Live at the Theatre Royal [2006].avi
deleting Dara O Brian/
deleting Daniel Tosh/Daniel Tosh - Completely Serious.divx
deleting Daniel Tosh/DVDCover.jpg
deleting Daniel Tosh/
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (5).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (4).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (3).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (2).avi
deleting Dane Cook/Insomniac Tour/Dane Cook - Insomniac Tour (1).avi
deleting Dane Cook/Insomniac Tour/
deleting Dane Cook/rough around the edges.avi
deleting Dane Cook/Stand up Comedy - Dane Cook on Kilborn - Lions.avi
deleting Dane Cook/Dane.Cook.Isolated.Incident.2009.DVDRip.XviD-Delta9.avi
deleting Dane Cook/DANEgerous Edition (iPod-ready).mp4
deleting Dane Cook/Comedy Central Presents - Dane Cook.mpg
deleting Dane Cook/Comedy Central Presents - Dane Cook, Harmful If Swallowed.avi
deleting Dane Cook/
deleting Dana Carvey/Dana Carvey - Critics' Choice (1995).avi
deleting Dana Carvey/
deleting Craig Ferguson/Craig Ferguson a wee bit o revolution 2009.avi
deleting Craig Ferguson/
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 5, 20-1).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 4, 40-21).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 3, 60-41).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 2, 80-61).avi
deleting Comedy central 100 greatest stand ups/Comedy Central's 100 Greatest Stand Ups (Part 1, 100-81).avi
deleting Comedy central 100 greatest stand ups/
deleting Comedy Central Roasts/The N.Y. Friars Club Roast of Drew Carey.avi
deleting Comedy Central Roasts/The N.Y. Friars Club Roast Of Hugh Hefner.mpg
deleting Comedy Central Roasts/The N.Y. Friars Club Roast Of Chevy Chase.avi
deleting Comedy Central Roasts/Comedy Central Roast of Pamela Anderson - Secret Stash Version.avi
deleting Comedy Central Roasts/Comedy Central Roast of Comedian Jeff Foxworthy.avi
deleting Comedy Central Roasts/Comedy Central Roast of Comedian Denis Leary.avi
deleting Comedy Central Roasts/Comedy Central Roast of Bob Saget.avi
deleting Comedy Central Roasts/Comedy Central Roast Of William Shatner 2006.avi
deleting Comedy Central Roasts/Comedy Central Roast Of Flavor Flav.avi
deleting Comedy Central Roasts/
deleting Comedy Central Presents/Comedy Central Presents - Zach Galifianakis.avi
deleting Comedy Central Presents/Comedy Central Presents - Wanda Sykes.avi
deleting Comedy Central Presents/Comedy Central Presents - Vince Morris.avi
deleting Comedy Central Presents/Comedy Central Presents - Vic Henley.avi
deleting Comedy Central Presents/Comedy Central Presents - Vanessa Hollingshead.avi
deleting Comedy Central Presents/Comedy Central Presents - Ty Barnett.avi
deleting Comedy Central Presents/Comedy Central Presents - Tracy Smith.avi
deleting Comedy Central Presents/Comedy Central Presents - Tony Woods.avi
deleting Comedy Central Presents/Comedy Central Presents - Tommy Johnagin.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Shillue.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Rhodes.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Papa.avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Papa(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tom Cotter.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Lynn.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Glass.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Barry.avi
deleting Comedy Central Presents/Comedy Central Presents - Todd Barry(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tig.avi
deleting Comedy Central Presents/Comedy Central Presents - The Sklar Brothers.mpg
deleting Comedy Central Presents/Comedy Central Presents - Tess.avi
deleting Comedy Central Presents/Comedy Central Presents - Ted Alexandro.avi
deleting Comedy Central Presents/Comedy Central Presents - Ted Alexandro(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Tammy Pescatelli.avi
deleting Comedy Central Presents/Comedy Central Presents - Sue Murphy.avi
deleting Comedy Central Presents/Comedy Central Presents - Steve McGrew.avi
deleting Comedy Central Presents/Comedy Central Presents - Steve Byrne.avi
deleting Comedy Central Presents/Comedy Central Presents - Stella.avi
deleting Comedy Central Presents/Comedy Central Presents - Slovin & Allen.avi
deleting Comedy Central Presents/Comedy Central Presents - Sheryl Underwood.avi
deleting Comedy Central Presents/Comedy Central Presents - Sebastian Maniscalco.avi
deleting Comedy Central Presents/Comedy Central Presents - Sean Cullen.wmv
deleting Comedy Central Presents/Comedy Central Presents - Scott Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Sabrina Matthews.avi
deleting Comedy Central Presents/Comedy Central Presents - Russell Peters.avi
deleting Comedy Central Presents/Comedy Central Presents - Russ Meneve.avi
deleting Comedy Central Presents/Comedy Central Presents - Rudy Rush.avi
deleting Comedy Central Presents/Comedy Central Presents - Rod Man.avi
deleting Comedy Central Presents/Comedy Central Presents - Rocky LaPorte.avi
deleting Comedy Central Presents/Comedy Central Presents - Robert Kelly.avi
deleting Comedy Central Presents/Comedy Central Presents - Robert Hawkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Rickey Smiley.avi
deleting Comedy Central Presents/Comedy Central Presents - Richard Jeni.avi
deleting Comedy Central Presents/Comedy Central Presents - Rich Vos.avi
deleting Comedy Central Presents/Comedy Central Presents - Retta.avi
deleting Comedy Central Presents/Comedy Central Presents - Reno Collier.avi
deleting Comedy Central Presents/Comedy Central Presents - Rene Hicks.avi
deleting Comedy Central Presents/Comedy Central Presents - Reggie McFadden.avi
deleting Comedy Central Presents/Comedy Central Presents - Ralph Harris.avi
deleting Comedy Central Presents/Comedy Central Presents - Peter Berman.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul Mercurio.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul Gilmartin.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul F.Tompkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Paul F. Tompkins(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Patrice O'Neal.avi
deleting Comedy Central Presents/Comedy Central Presents - Pat Dixon.avi
deleting Comedy Central Presents/Comedy Central Presents - Otis Lee Crenshaw.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Swardson.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Swardson(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Nick Dipaolo.avi
deleting Comedy Central Presents/Comedy Central Presents - Nick DiPaolo(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Mitch Hedburg.WMV
deleting Comedy Central Presents/Comedy Central Presents - Mitch Hedburg uncut.avi
deleting Comedy Central Presents/Comedy Central Presents - Mitch Fatel.avi
deleting Comedy Central Presents/Comedy Central Presents - Mike Britt.wmv
deleting Comedy Central Presents/Comedy Central Presents - Mike Birbiglia.avi
deleting Comedy Central Presents/Comedy Central Presents - Mike Birbiglia(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Megan Mooney.avi
deleting Comedy Central Presents/Comedy Central Presents - Mark Curry.avi
deleting Comedy Central Presents/Comedy Central Presents - Mario Cantone.avi
deleting Comedy Central Presents/Comedy Central Presents - Maria Bamford.wmv
deleting Comedy Central Presents/Comedy Central Presents - Maria Bamford(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Margaret Smith.avi
deleting Comedy Central Presents/Comedy Central Presents - Marc Maron.avi
deleting Comedy Central Presents/Comedy Central Presents - Marc Maron(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Lynne Koplitz.avi
deleting Comedy Central Presents/Comedy Central Presents - Louis Ramey.avi
deleting Comedy Central Presents/Comedy Central Presents - Loni Love.avi
deleting Comedy Central Presents/Comedy Central Presents - Lizz Winstead.avi
deleting Comedy Central Presents/Comedy Central Presents - Lisa Landry.avi
deleting Comedy Central Presents/Comedy Central Presents - Lenny Clarke.avi
deleting Comedy Central Presents/Comedy Central Presents - Laura Kightlinger.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Grooms.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Dunnigan.avi
deleting Comedy Central Presents/Comedy Central Presents - Kyle Cease.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Nealon.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Hart.avi
deleting Comedy Central Presents/Comedy Central Presents - Kevin Brennan.avi
deleting Comedy Central Presents/Comedy Central Presents - Keith Robinson.avi
deleting Comedy Central Presents/Comedy Central Presents - Kathleen Madigan.avi
deleting Comedy Central Presents/Comedy Central Presents - Judy Gold.avi
deleting Comedy Central Presents/Comedy Central Presents - Josh Sneed.avi
deleting Comedy Central Presents/Comedy Central Presents - Jon Reep.avi
deleting Comedy Central Presents/Comedy Central Presents - John Heffron.divx
deleting Comedy Central Presents/Comedy Central Presents - John Heffron(2).avi
deleting Comedy Central Presents/Comedy Central Presents - John Caparulo.avi
deleting Comedy Central Presents/Comedy Central Presents - Joey Kola.avi
deleting Comedy Central Presents/Comedy Central Presents - Joe Matarese.avi
deleting Comedy Central Presents/Comedy Central Presents - Jo Koy.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Shubert.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Pardo.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Dore.avi
deleting Comedy Central Presents/Comedy Central Presents - Jimmy Carr.avi
deleting Comedy Central Presents/Comedy Central Presents - Jim Gaffigan.asf
deleting Comedy Central Presents/Comedy Central Presents - Jim David.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeremy Hotz.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeff Stilson.mpg
deleting Comedy Central Presents/Comedy Central Presents - Jeff Garcia.avi
deleting Comedy Central Presents/Comedy Central Presents - Jeff Dunham.mpg
deleting Comedy Central Presents/Comedy Central Presents - Jeff Cesario.avi
deleting Comedy Central Presents/Comedy Central Presents - Jamie Lissow.avi
deleting Comedy Central Presents/Comedy Central Presents - Jake Johannsen.avi
deleting Comedy Central Presents/Comedy Central Presents - Jackie Kashian.avi
deleting Comedy Central Presents/Comedy Central Presents - Jack Coen.avi
deleting Comedy Central Presents/Comedy Central Presents - Ian Bagg.avi
deleting Comedy Central Presents/Comedy Central Presents - Hugh Fink.avi
deleting Comedy Central Presents/Comedy Central Presents - Howard Kremer.avi
deleting Comedy Central Presents/Comedy Central Presents - Henry Phillips.avi
deleting Comedy Central Presents/Comedy Central Presents - Harland Williams.avi
deleting Comedy Central Presents/Comedy Central Presents - Hard'N'Phirm.avi
deleting Comedy Central Presents/Comedy Central Presents - Gregg Rogell.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Proops.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Giraldo.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Giraldo(2).avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Fitzsimmons[2].avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Fitzsimmons.avi
deleting Comedy Central Presents/Comedy Central Presents - Greg Behrendt.avi
deleting Comedy Central Presents/Comedy Central Presents - Godfrey.avi
deleting Comedy Central Presents/Comedy Central Presents - Gilbert Gottfried.avi
deleting Comedy Central Presents/Comedy Central Presents - Gene Pompa.avi
deleting Comedy Central Presents/Comedy Central Presents - Gary Valentine.avi
deleting Comedy Central Presents/Comedy Central Presents - Freddy Soto.avi
deleting Comedy Central Presents/Comedy Central Presents - Frank Caliendo.wmv
deleting Comedy Central Presents/Comedy Central Presents - Finesse Mitchell.avi
deleting Comedy Central Presents/Comedy Central Presents - Eugene Mirman.avi
deleting Comedy Central Presents/Comedy Central Presents - Elvira Kurt.avi
deleting Comedy Central Presents/Comedy Central Presents - Eddie Gossling.avi
deleting Comedy Central Presents/Comedy Central Presents - Eddie Brill.avi
deleting Comedy Central Presents/Comedy Central Presents - Earthquake.avi
deleting Comedy Central Presents/Comedy Central Presents - Dwayne Perkins.avi
deleting Comedy Central Presents/Comedy Central Presents - Dwayne Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Drew Fraser.avi
deleting Comedy Central Presents/Comedy Central Presents - Dov Davidoff.avi
deleting Comedy Central Presents/Comedy Central Presents - Doug Stanhope.avi
deleting Comedy Central Presents/Comedy Central Presents - Doug Benson.avi
deleting Comedy Central Presents/Comedy Central Presents - Don DC Curry.mpg
deleting Comedy Central Presents/Comedy Central Presents - Dom Irrera.avi
deleting Comedy Central Presents/Comedy Central Presents - Deon Cole.avi
deleting Comedy Central Presents/Comedy Central Presents - DeRay.avi
deleting Comedy Central Presents/Comedy Central Presents - David Feldman.avi
deleting Comedy Central Presents/Comedy Central Presents - Dat Phan.avi
deleting Comedy Central Presents/Comedy Central Presents - Darrell Hammond.avi
deleting Comedy Central Presents/Comedy Central Presents - Daniel Tosh.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Naturman.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Mintz.avi
deleting Comedy Central Presents/Comedy Central Presents - Dan Cummins.avi
deleting Comedy Central Presents/Comedy Central Presents - D.C. Benny.divx
deleting Comedy Central Presents/Comedy Central Presents - Craig Shoemaker.avi
deleting Comedy Central Presents/Comedy Central Presents - Corey Kahaney.avi
deleting Comedy Central Presents/Comedy Central Presents - Corey Holcomb.avi
deleting Comedy Central Presents/Comedy Central Presents - Clinton Jackson.avi
deleting Comedy Central Presents/Comedy Central Presents - Christian Finnegan.avi
deleting Comedy Central Presents/Comedy Central Presents - Chelsea Handler.avi
deleting Comedy Central Presents/Comedy Central Presents - Carol Leifer.avi
deleting Comedy Central Presents/Comedy Central Presents - Carlos Alazraqui.avi
deleting Comedy Central Presents/Comedy Central Presents - Bruce Bruce.avi
deleting Comedy Central Presents/Comedy Central Presents - Brian Regan.mpg
deleting Comedy Central Presents/Comedy Central Presents - Bob Oschack.avi
deleting Comedy Central Presents/Comedy Central Presents - Bob Marley.avi
deleting Comedy Central Presents/Comedy Central Presents - Bill Santiago.avi
deleting Comedy Central Presents/Comedy Central Presents - Bill Burr.avi
deleting Comedy Central Presents/Comedy Central Presents - Bil Dwyer.avi
deleting Comedy Central Presents/Comedy Central Presents - Big Jay Oakerson.avi
deleting Comedy Central Presents/Comedy Central Presents - Ben Bailey.avi
deleting Comedy Central Presents/Comedy Central Presents - Arnez J..avi
deleting Comedy Central Presents/Comedy Central Presents - Arj Barker[2].avi
deleting Comedy Central Presents/Comedy Central Presents - Arj Barker.avi
deleting Comedy Central Presents/Comedy Central Presents - Aries Spears.avi
deleting Comedy Central Presents/Comedy Central Presents - Ardal O'Hanlon.avi
deleting Comedy Central Presents/Comedy Central Presents - Andy Kindler.avi
deleting Comedy Central Presents/Comedy Central Presents - Andrew Kennedy.avi
deleting Comedy Central Presents/Comedy Central Presents - Alonzo Bodden.avi
deleting Comedy Central Presents/Comedy Central Presents - Al Madrigal.avi
deleting Comedy Central Presents/Comedy Central Presents - Adele Givens.wmv
deleting Comedy Central Presents/Comedy Central Presents - Adam Ferrara.avi
deleting Comedy Central Presents/.DS_Store
deleting Comedy Central Presents/
deleting Chubby Brown/Chubby Brown - Bad Taste [2003].avi
deleting Chubby Brown/
deleting Chris Rock/Chris Rock Big Ass Jokes.avi
deleting Chris Rock/Chris Rock - Never Scared [2004].avi
deleting Chris Rock/Chris Rock - Kill the messenger.avi
deleting Chris Rock/Chris Rock - Bring the pain 1996.divx
deleting Chris Rock/Chris Rock - Bigger And Blacker (2002).avi
deleting Chris Rock/
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/Extras/fico-cmtbofif-deleted-scenes.avi
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/Extras/
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/fico-cmtbofif.nfo
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/fico-cmtbofif.avi
deleting Carlos Mencia/Carlos Mencia The Best Of Funny Is Funny 2007/
deleting Carlos Mencia/carlos mencia3.avi
deleting Carlos Mencia/Comedy Central Presents - Carlos Mencia.mpg
deleting Carlos Mencia/Carlos Mencia2.divx
deleting Carlos Mencia/Carlos Mencia.divx
deleting Carlos Mencia/
deleting Brian Regan/Brian Regan - Live Mp3/playlist.m3u
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - You Too Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Whale Noises.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Stupid in School.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Spider Webs Bees.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Seatbelts.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Peanut Butter Jelly.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Monster Truck Drivers.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Lousy in Little League.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Log Trucks.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Kids Party Games.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Horns Windshields.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Hooked on Phonix.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Health Club Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Flipper Gentle Ben.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Fishing on TV.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Evel Knievel.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Elevators Faces.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Donut Lady.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Dog Barking.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Crank Calls.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Blasting Zone.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Big Family Stuff.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Belly Buttons.mp3
deleting Brian Regan/Brian Regan - Live Mp3/Brian Regan - Animals.mp3
deleting Brian Regan/Brian Regan - Live Mp3/
deleting Brian Regan/Brian Regan - Standing Up.avi
deleting Brian Regan/Brian Regan - I Walked on the Moon.avi
deleting Brian Regan/Brian Regan - Epitome Of Hyperbole.avi
deleting Brian Regan/
deleting Brendon Burns/brendon.burns.so.i.suppose.this.is.offensive.now.2008.ws.dvdrip.xvid-pvr.nfo
deleting Brendon Burns/Brendon Burns - So I Suppose THIS Is Offensive Now.avi
deleting Brendon Burns/
deleting Bonnie McFarlane/Bonnie McFarlane - One Night Stand.avi
deleting Bonnie McFarlane/
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-51/rsync/rsync.c(244) [sender=2.6.9]

THANKS MAN THAT TOOK A SECOND BEFORE I REALISED WHAT HAPPENED NO WARNING ALL THAT DATA GONE...
@casmarc

This comment has been minimized.

Sign in to view
Copy link

@casmarc casmarc commented Dec 16, 2016

That why you dont just paste anything you read on the internet in your terminal. LOL
@pando85

This comment has been minimized.

Sign in to view
Copy link

@pando85 pando85 commented Dec 24, 2016

To allow arcfour cipher:

sshd config

Add at the end of /etc/sshd_config file:

# enable all ciphers!
# obtained with ssh -Q cipher localhost | paste -d , -s
Ciphers 3des-cbc,blowfish-cbc,cast128-cbc,arcfour,arcfour128,arcfour256,aes128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
@claytondukes

This comment has been minimized.

Sign in to view
Copy link

@claytondukes claytondukes commented Jan 4, 2017

I'm only getting marginally better transfers. Would anyone happen to know why?

here's what I have:

# Rsync options:
src='user@host:/dir/'
dest='/dir/'
excludefile='/root/.ignore-sync'
deleteOld="--delete-before --delete-excluded --max-delete=50"
options=(-aHAXxv --progress --partial --append --stats --min-size=1 --prune-empty-dirs --numeric-ids ${deleteOld} --exclude-from=${excludefile} ${limit} -e 'ssh -T -c arcfour -o Compression=no -x')

rsync "${options[@]}" ${src} ${dest}

Prior to using your settings, I used:

syncCmd="rsync -rltvhm --progress --partial --append --stats --min-size=1 --prune-empty-dirs ${deleteOld} --exclude-from=${excludefile} ${limit} ${src} ${dest}"

Before using yours I got around 4mbps
now I get around 6mbps.
As you can see below, I should have plenty of bandwidth.

Link speedtests
Host (Upload Speed used here):

# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Google Fiber (136.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Google Fiber: 18.029 ms
Testing download speed................................................................................
Download: 477.76 Mbit/s
Testing upload speed....................................................................................................
Upload: 173.21 Mbit/s

Client (Download Speed used here):

# speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from XFINITY (73.x.x.x)...
Selecting best server based on latency...
Hosted by CenturyLink, Inc: 47.933 ms
Testing download speed........................................
Download: 85.72 Mbit/s
Testing upload speed..................................................
Upload: 11.77 Mbit/s
@thesandbender

This comment has been minimized.

Sign in to view
Copy link

@thesandbender thesandbender commented Mar 21, 2017 •
edited

arcfour (RC4) is disabled for a reason. There are known, working exploits against it that allow RC4 encrypted data to be deciphered. If you have a AMD or Intel CPU that was made after 2010-2011 it probably supports AES-NI and you're better off using that as @aflyhorse suggested. You should never intentionally weaken a system's security, even if "it's just for a little bit".
@henryx

This comment has been minimized.

Sign in to view
Copy link

@henryx henryx commented May 25, 2017

Use -P instead --progress, because this is equivalent for --progress --partial
@toloveru

This comment has been minimized.

Sign in to view
Copy link

@toloveru toloveru commented Jun 17, 2017 •
edited

Thanks a lot! Backup speeds to my local server improved significantly! Now it hits 60 MB/s easily 😀
@dukelambert

This comment has been minimized.

Sign in to view
Copy link

@dukelambert dukelambert commented Jun 28, 2017

Really thanks for this post, the transfer rate is improved up to 90 MB/s . I'd like to know how to place '--exclude /backup/somedir' in the string because it doesn't work. Thanks in advance.
@ktavabi

This comment has been minimized.

Sign in to view
Copy link

@ktavabi ktavabi commented Aug 20, 2017

Thanks for this! I hit >60 MB/S rates without the weak SSH cipher. Don't think -c arcfour is really necessary.
@denysvitali

This comment has been minimized.

Sign in to view
Copy link

@denysvitali denysvitali commented Nov 26, 2017

@HaniKali Same here ahah. That's why you shouldn't copy & paste random commands from the internet. I assumed --delete was used to delete remote files once they were synced. I "lost" 200+ GB of music files on my Plex server, but I have a backup, so it isn't really a big deal 🤣
@tomrunia

This comment has been minimized.

Sign in to view
Copy link

@tomrunia tomrunia commented Nov 29, 2017

Since -c arcfour is considered obsolete, using -c aes128-gcm@openssh.com is probably fastest.

rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" [source_directory] user@hostname:[target_directory]/

See these encryption benchmark results: https://famzah.files.wordpress.com/2015/06/openssh-ciphers-performance-2015-chart.jpg
@janmd

This comment has been minimized.

Sign in to view
Copy link

@janmd janmd commented Dec 6, 2017

Hi,

currently transferring from one server to another of 1.5TB. I use this command

rsync -av --progress /home/user/public_html/* :/home/user/public_html/

I started it around 6hrs ago. Source server is having 100Mbit port speed. Destination is having 1Gbit. I issue the command from the source server.

I checked my server output bandwidth and it reports average 25Mbps for the last 4hrs

Is it still worth stopping it and using this command?

Also, I have no existing backup. Will this command remove the source? Im a bit worried because of the post by HaniKali.

This is my first time using rsync and I don't want to lose anything from source.

Can someone confirm if this will not delete my source? What happened to HaniKali? Why his files was deleted by using the command?

Thanks
@okhosting

This comment has been minimized.

Sign in to view
Copy link

@okhosting okhosting commented Dec 7, 2017

Very usefull post, thanks!
@genokan

This comment has been minimized.

Sign in to view
Copy link

@genokan genokan commented Dec 9, 2017

@janmd..

Read the man page to understand what each thing is doing.

   --delete
          This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized.  You must have asked
          rsync  to  send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus
          gets a request to transfer individual files, not the files’ parent directory.  Files that are excluded from the transfer  are  also  excluded  from  being  deleted  unless  you  use  the
          --delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).

          Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled.  Beginning with 2.6.7, deletions will also occur when --dirs (-d) is enabled, but only for directo‐
          ries whose contents are being copied.

          This option can be dangerous if used incorrectly!  It is a very good idea to first try a run using the --dry-run option (-n) to see what files are going to be deleted.

          If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. This is to prevent temporary filesystem  failures  (such  as
          NFS errors) on the sending side from causing a massive deletion of files on the destination.  You can override this with the --ignore-errors option.

          The  --delete  option  may be combined with one of the --delete-WHEN options without conflict, as well as --delete-excluded.  However, if none of the --delete-WHEN options are specified,
          rsync will choose the --delete-during algorithm when talking to rsync 3.0.0 or newer, and the --delete-before algorithm when talking to an  older  rsync.   See  also  --delete-delay  and
@alinz

This comment has been minimized.

Sign in to view
Copy link

@alinz alinz commented Jan 20, 2018

Just want to say awesome. Thank you, I manage to upload really fast on very slow connection!
@felixlindemann

This comment has been minimized.

Sign in to view
Copy link

@felixlindemann felixlindemann commented Feb 14, 2018 •
edited

Thank you! made my Day. Improved speed by the factor of 8.5 :-D
I had to modify it slightly to my setup qnap (new device) wdmycloud(old device) as the encryption wasn't accepted

you can find my version here
rsync -rtXx --numeric-ids --progress -e "ssh -T -c aes128-ctr -o Compression=no -x" <user>@<ip>:/<source-path>/ /<dest.path>/
https://gist.github.com/felixlindemann/dfee61a20bbd838cdb26f44cc5552404
@jjuanmera

This comment has been minimized.

Sign in to view
Copy link

@jjuanmera jjuanmera commented Mar 22, 2018

I was looking a way to speed up the transfer between mac os and Raspberry pi. This message is after input the password:

rsync: on remote machine: -vlHogDtpAXrxe.iLsfx: unknown option
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(1333) [server=2.6.9]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

Any ideas? Thanks
@tufahu

This comment has been minimized.

Sign in to view
Copy link

@tufahu tufahu commented May 3, 2018

thank you! very good hints!
@win4me

This comment has been minimized.

Sign in to view
Copy link

@win4me win4me commented May 10, 2018 •
edited

Comparing the methods in dust-free lab over the 10 Gbit "jumbo frame" network link:

-   13.6 MB/s  rsync -az 
-   16.7 MB/s  scp -Cr
-   44.8 MB/s  rsync -a 
-   59.8 MB/s  sftp
-   61.2 MB/s  scp -r 
-   61.4 MB/s  sftp  -R 128 -B 65536
-   62.4 MB/s  rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"
-  143.5 MB/s  scp -r -c arcfour
-  144.2 MB/s  sftp -oCiphers=arcfour

Read details of the benchmark test, commands, and output in my post: http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html
@teusbenschop

This comment has been minimized.

Sign in to view
Copy link

@teusbenschop teusbenschop commented Jun 6, 2018

Thank you, the speedup makes a good difference!
@danielmotaleite

This comment has been minimized.

Sign in to view
Copy link

@danielmotaleite danielmotaleite commented Jul 12, 2018

If you can, please put the --delete descripiton in bold and with a bigger warming, so people like HaniKali so not remove delete valid content.

There are always people that just copy&paste, with bigger warnings is totally their fault!
@Cho1409

This comment has been minimized.

Sign in to view
Copy link

@Cho1409 Cho1409 commented Jul 13, 2018

Witchcraft! Thank you!
@areeb111

This comment has been minimized.

Sign in to view
Copy link

@areeb111 areeb111 commented Aug 6, 2018

Thank you, I got 110MB/s same as your command but only with rsync -av and ssh attributes
@seandex

This comment has been minimized.

Sign in to view
Copy link

@seandex seandex commented Aug 22, 2018

is arcfour even possible in 2018?

cat /proc/cpuinfo | grep aes
if you have it then use AES ciphers.
moreover, if both-end have it, you can achieve it up to 900Mbps.
@harish2704

This comment has been minimized.

Sign in to view
Copy link

@harish2704 harish2704 commented Aug 23, 2018

Thanks for sharing this information.

I tried this and this is what I understood.

  • Network bandwidth utilization doesn't represent amount of data transferred.
  • When I was using this script, I got 90Mb/s network usage and 11MB/s data transfer.
  • When I removed -o compression=no from ssh , data transfer increased to 20MB/s
  • When I enabled z option in rsync datatransfer rate increased to 29MB/s ( and network utilization was 60Mb/s )

Lesson learned:

  • This script may be helpful to when we have un-compressible data and we have Gigabit NIC.
  • but when network becomes bottleneck and/or your data is compressible , this method may not work well.
@scottinan

This comment has been minimized.

Sign in to view
Copy link

@scottinan scottinan commented Sep 13, 2018

Thanks! From an SSD mdadm array on ext4 to a ZFS raid z2 with 2TBs spinning disks I'm getting maxing out at 1GB lol gonna have to set up a bonded interface! 👍
@sjuxax

This comment has been minimized.

Sign in to view
Copy link

@sjuxax sjuxax commented Sep 22, 2018

is arcfour even possible in 2018?

cat /proc/cpuinfo | grep aes
if you have it then use AES ciphers.
moreover, if both-end have it, you can achieve it up to 900Mbps.

arcfour is not compiled in to my OpenSSH daemon. chacha20-poly1305@openssh.com is the modern replacement. I get ~100MB/s with that v. capping out around 40MB/s with default ciphers (AES).
@88plug

This comment has been minimized.

Sign in to view
Copy link

@88plug 88plug commented Oct 1, 2018 •
edited

rsync -av --progress -e "ssh -T -c aes128-ctr -o Compression=no -x" source/ user@x.x.x.x:/destination

The key is leaving off the compression (-z) and using aes128-ctr on Ubuntu 18 (latest cipher support)
@MBetters

This comment has been minimized.

Sign in to view
Copy link

@MBetters MBetters commented Dec 4, 2018 •
edited

I'm using this script. It gets rid of the extended attributes ("-X" flag) if you're running this on Windows. It also uses the aes128-gcm cipher in the likely case that your openssh installation doesn't include arcfour.

#!/bin/bash
# Fast rsync command

# Set the RSYNC_ARGS.
UNAME="$(uname -s)"
case "${UNAME}" in 
    Linux* | Darwin*)
        RSYNC_ARGS="-aHAXxv --numeric-ids --delete --progress -e"
    ;;
    # Windows filesystems do not support extended attributes (the "-X" option)
    CYGWIN* | MINGW*)
        RSYNC_ARGS="-aHAxv --numeric-ids --delete --progress -e"
    ;;
    *)
        echo "ERROR: Running on unknown system! Exiting!"
        return 1
    ;;
esac

# Set the SSH_ARGS
SSH_ARGS="-T -o -c aes128-gcm@openssh.com Compression=no -x"

# Get the rest of the args from the caller
USER=$1
SOURCE=$2
SOURCE_DIR=$3
DEST_DIR=$4

rsync $RSYNC_ARGS "ssh $SSH_ARGS" $USER@$SOURCE:$SOURCE_DIR $DEST_DIR
@rfjakob

This comment has been minimized.

Sign in to view
Copy link

@rfjakob rfjakob commented May 11, 2019 •
edited

I benchmarked this a little bit with 1GB of random data, modern software (Fedora 30) on a CPU that does NOT have AES acceleration. Test data:

dd if=/dev/urandom of=/tmp/1g bs=1M count=1024

rsync default settings -> 22.71MB/s

rsync -P /tmp/1g 127.0.0.1:/tmp/1g.2

disably tty allocation -> 20.39MB/s

rsync -P /tmp/1g -e "ssh -T" 127.0.0.1:/tmp/1g.2

disable compression -> 193.42MB/s

rsync -P /tmp/1g -e "ssh -o Compression=no" 127.0.0.1:/tmp/1g.2

scp -> 109.9MB/s

scp -o Compression=no /tmp/1g 127.0.0.1:/tmp/1g.2

Conclusion

Disable compression, but don't bother with arcfour, even without AES acceleration you'll be faster that gigabit ethernet. Disabling tty allocation had no effect in my testing. Scp is significantly slower than rsync.

Running ssh -v SERVER you'll what cipher is used:

debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none

So we are using aes256-gcm, which fortunately is the same cipher that gocryptfs uses, so we can look at this benchmark table: https://github.com/rfjakob/gocryptfs/wiki/CPU-Benchmarks . You should basically get >100MB/s on any x86 CPU younger than 10 years.
@deajan

This comment has been minimized.

Sign in to view
Copy link

@deajan deajan commented Aug 14, 2019 •
edited

3fr/3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/arw/asf/avi/bz2/cab/cr2/crypt[5678]/dat/dcr/deb/dmg/drc/ear/erf/flac/flv/gif/gpg/gz/iiq/iso/jar/jp2/jpeg/jpg/k25/kdc/lz/lzma/lzo/m4[apv]/mef/mkv/mos/mov/mp[34]/mpeg/mp[gv]/msi/nef/oga/ogg/ogv/opus/orf/pef/png/qt/rar/rpm/rw2/rzip/s7z/sfx/sr2/srf/svgz/t[gb]z/tlz/txz/vob/wim/wma/wmv/xz/zip

Added some more extensions to that list:

3fr/3g2/3gp/3gpp/7z/aac/ace/amr/apk/appx/appxbundle/arc/arj/arw/asf/avi/bz/bz2/cab/cr2/crypt[5678]/dat/dcr/deb/dmg/drc/ear/erf/flac/flv/gif/gpg/gz/iiq/jar/jp2/jpeg/jpg/h26[45]/k25/kdc/kgb/lha/lz/lzma/lzo/lzx/m4[apv]/mef/mkv/mos/mov/mp[34]/mpeg/mp[gv]/msi/nef/oga/ogg/ogv/opus/orf/pak/pef/png/qt/rar/r[0-9][0-9]/rz/rpm/rw2/rzip/sfark/sfx/s7z/sr2/srf/svgz/t[gb]z/tlz/txz/vob/wim/wma/wmv/xz/zip
@vrossum

This comment has been minimized.

Sign in to view
Copy link

@vrossum vrossum commented Aug 26, 2019

I benchmarked this a little bit with 1GB of random data

On random data compression can't help. However, if you have text files and a slow connection, it is a logical choice.
@maxnoe

This comment has been minimized.

Sign in to view
Copy link

@maxnoe maxnoe commented Aug 26, 2019

zst is also worth adding
@slmingol

This comment has been minimized.

Sign in to view
Copy link

@slmingol slmingol commented Sep 5, 2019

SSH_ARGS="-T -o -c aes128-gcm@openssh.com Compression=no -x"

you have a typo here, that should be like this:

SSH_ARGS="-T -c aes128-gcm@openssh.com -o Compression=no -x"
@slmingol

This comment has been minimized.

Sign in to view
Copy link

@slmingol slmingol commented Sep 5, 2019

Saw not much movement in the variations here when testing across 2 ec2 instances on AWS in 2 different AZs. I ended up using this which includes most of what I could gleam as optimizations but the speed fluctuated enough among with and without that your mileage will definitely vary as well.

$ rsync -aHAxv --numeric-ids --delete -P -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" 500MB.file 10.16.87.187:~
@teran-mckinney

This comment has been minimized.

Sign in to view
Copy link

@teran-mckinney teran-mckinney commented Sep 26, 2019 •
edited

Regarding the suggested -T, there is no TTY allocation by default when SSH runs a comand on the host vs running a shell. It's redundant. On some systems (Debian 9 and later, maybe some earlier), you can do -c none which is ideal if you care about speed and not privacy.
@danielmotaleite

This comment has been minimized.

Sign in to view
Copy link

@danielmotaleite danielmotaleite commented Oct 7, 2019 •
edited

i have done a small up-to-date ssh test using several cipher, between 2 AWS r5.12xlarge and got this:

chacha20-poly1305@openssh.com        190.89MB/s  (default if not option used)
aes128-ctr                           259.01MB/s
aes256-gcm@openssh.com               339.05MB/s
aes128-gcm@openssh.com               298.20MB/s
none                                 189.72MB/s

i didn't test arcfour, but in previous tests, it was faster... but as require changing the sshd server to support that cipher, i'm trying to avoid it
interesting is the aes256-gcm is faster than the aes128-gcm, probably because of optimization and hardware support. the cipher none, while it do not return error, seems to fallback to the default, so anyone saying that -c none will disable encryption probably do not know that its really using the default cipher! :)

No other ciphers were tested as current ssh only have those as default enabled ciphers
@sasha2002

This comment has been minimized.

Sign in to view
Copy link

@sasha2002 sasha2002 commented May 22, 2020

To all.

I'd like to know how to place '--exclude /backup/somedir' in the string because it doesn't work. Thanks in advance.

To exclude a directory("<source_dir>/bigDir") you need to put the name of directory in current directory like this example :
rsync -aHAXxv --numeric-ids --delete --progress --exclude 'bigDir' -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" user@:<source_dir> <dest_dir>
@davidbitton

This comment has been minimized.

Sign in to view
Copy link

@davidbitton davidbitton commented May 25, 2020

for a Mac to Linux transfer it's useful to use other options. Arcfour is not available for most new machines anymore and UTF-8 on OS X is different than UTF-8 on Linux (important if you have Umlauts like Germans, Samba/NFS will fail otherwise). My command if both (Mac & Linux) machines support AES on their processors and you want to transfer from Mac to Linux:

rsync -rltv --progress --human-readable --delete --iconv=utf-8-mac,utf-8 -e 'ssh -T -c aes128-gcm@openssh.com -o Compression=no -x' <local_mac_source> <remote_linux_dest>

reverse the iconv option if you want to transfer from Linux to Mac.

how would you do mac to mac?
@L1so

This comment has been minimized.

Sign in to view
Copy link

@L1so L1so commented Jun 3, 2020

Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.
@brianlamb

This comment has been minimized.

Sign in to view
Copy link

@brianlamb brianlamb commented Jun 28, 2020

This was such a great post to find!
I was set to leave my transfer going at 5-10MB/s but couldn't go to sleep with 1.2TB going for 30hours!
(This was also from Synology NAS to MacOS)

As others mentioned early on in this post using specific SSH options can affect the transfer rate dramatically: -e "ssh -T -c aes128-ctr -o Compression=no -x". Primarily the Compression factor. I couldn't see notable differences and didn't test more than comparing to "-c aes256-gcm@openssh.com" but got variably up to 50-90MB/sec.

DO use --dry-run and --itemize-changes which is a great record of what is actually going to happen.
Always be careful of SOURCE and DESTINATION.
Pause and think, before setting things in motion!

If you want a little help managing a collection of commands you run and an environment conducive to setting up rsync command lines you could try (on a Mac) RsyncOSX as a GUI front end (although I still prefer to run the actual command in a standalone terminal.)
@pricesgoingup

This comment has been minimized.

Sign in to view
Copy link

@pricesgoingup pricesgoingup commented Aug 12, 2020

@L1so > Not recommended, I almost trashed my entire movies collection by doing this, good thing I canceled it.

Don't paste everything you see on the internet without looking the flags up first /shrugs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment