Quicky Transferring large viles over high-latency WAN link using UDR/UDT

UDR-UDT.md

Motivation

At some point, you might find yourself in need to transfer a file (i.e. a dump of the database, or a backup tarball) quickly over a WAN link. Unfortunately, regular copy speeds are impaired by TCP protocol and latency and speed of light - and link bandwidth has very limited effect on the transfer speeds. However, there is a workaround: one can use UDP-based file transfer (UDT). Here is a proof-of-concept experiment that demonstrates the transfer speed improvement.

Experiment

Create a large file to transfer:

localhost~$ dd if=/dev/zero of=10G-bulk.bin bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 41.4086 s, 259 MB/s
$ du -sm *
10241   10G-bulk.bin

Baseline: time SCP transfer at 25MBps:

localhost~$ time scp 10G-bulk.bin faraway.system.local:
10G-bulk.bin                                                                                                                                                    100%   10GB  25.9MB/s   06:34
 
real    6m35.970s
user    0m47.704s
sys 0m30.804s

Install udr and rsync on both servers

see https://github.com/martinetd/UDR

git clone https://github.com/martinetd/UDR
cd UDR
make
strip src/udr
scp src/udr localhost~:
scp src/udr faraway:

Cross-compiling in docker

Note that if you are using an OS version differnt from the target, you might need to use docker. Here is how you cross-compile ofr Debian9:

localhost~$ cat  > Dockerfile << EOF
FROM debian:stretch
  
RUN apt update && apt install -y curl build-essential cmake git g++ pkg-config libssl-dev
ENTRYPOINT /bin/bash
EOF
 
$ docker build . --tag deb9
$ docker run -it -v `pwd`:/root/src deb9
container~# cd /root/src && make
# exit
$ strip src/udr
$ scp src/udr server1:
$ scp src/udr faraway:

time UDR transfer for the same file. note the 5x speed increase (125Mbps)

• Your mileage may vary. Speed up depends on link latency - I experimented with ~60ms link
• Make sure that UDP ports are open. This may, or may not be the case. In may systems, dynamic port range (49152 to 65535) is open

localhost~$ ./udr -c ./udr -a 60000 -b 61000 rsync -av --stats --progress 10G-bulk.bin faraway.system.local:
sending incremental file list
10G-bulk.bin
 10,737,418,240 100%  125.11MB/s    0:01:25 (xfr#1, to-chk=0/1)

See also

• UDR/UDT: https://ciara.fiu.edu/images/udr_poster_sc12.pdf:
  • UDT is a UDP-based, application level protocol. It is designed to support transferring large datasets over high speed wide area networks, where TCP has been known to be extremely ineffective. UDT’s features include:
  • UDP-based, application level protocol
  • Protocol design to support efficient packet processing
  • Configurable congestion control
  • Efficient native congestion control algorithm
  • Optimized implementation as a user level C++ library
  • Supports Linux, BSD, UNIX, and Windows
  • API very similar to BSD Socket
  • Open source BSD license