Efficient file copying using pigz and nc

copy_files_efficiently.txt

From Tumblr slidedeck "Massively Sharded MySQL" (http://assets.en.oreilly.com/1/event/74/Massively%20Sharded%20MySQL%20at%20Tumblr%20Presentation.pdf):

Our shard-split process relies on creating new slaves quickly, which involves copying around very large data sets.

For compression we use pigz (parallel gzip), but there are other alternatives. 

On destination box:

$ nc -l [port] | pigz -d | tar xvf -

On source box:

$ tar vc . | pigz | nc [destination hostname] [port]