Faster UDP file transfers using tsunami

tsunami.MD

I recently had to move about 3TB of video footage between residential gigabit fiber connections. The fastest speed I could get was around ~150Mbps through Dropbox and other cloud providers, so I opted for a more direct answer.

After using Resilio Sync to buffer the files onto an Ubuntu Virtual Machine, I installed Tsunami. Using this I was able to get 600Mbps with no tuning, between a server in Los Angeles to my residential connection in Pittsburgh.

tsunami

What it does:

• Very fast file transfers because it's UDP based

What it does not:

• Support the ability to transfer more than one file [1]
• Support the abilty to crawl through subdirectories
• Support encryption

Those first two points were a killer for me, so I wrote a litte script to crawl every file in a folder.

Installation

These steps have been tested on Ubuntu 16.04. You'll need to do these steps both on your source and your destination servers.

1. Clone this forked repository which has recent bug fixes, and then navigate to the working directory

git clone https://github.com/sebsto/tsunami-udp.git
cd tsunami-udp

2. Install compiling tools if you haven't yet:

sudo apt-get install build-essential libtool automake autoconf

3. Compile the program

./recompile.sh
make
sudo make install

Preparing the source server

On your source server, navigate to a directory containing files you want to download, and then create a list of the files (but not folders) that it contains.

cd path/to/my/files
ls -1ap | grep -v /

Copy the list of files to your clipboard, and then launch the tsunami daemon. You may choose to run it in screen because after this we can disconnect from the source server.

tsunamid

Preparing the destination server

On your destination server, create the following file and save it somewhere in your PATH (type echo $PATH to see a list of your choices). I named mine tsdl.

nano /path/to/your/script/tsdl

Paste the following code in:

#!/bin/bash

cat $1 | while read line || [[ -n $line ]];
do
   echo "trying $line"
   tsunami connect your_source_ip_or_hostname_here get $line quit
done

Replace your_source_ip_or_hostname_here with the IP address or hostname of your source server, and save + close. Then make your script executable:

chmod +x /path/to/your/script/tsdl

Starting the transfer

On your destination server, navigate to the directory where you want the files to download to. Then create a file containing the filenames you want to download - I'll call mine list.txt.

cd /path/to/download/files/to
nano list.txt

Paste in the list of filenames from your clipboard from earlier, save the file and close, and now you're ready to run tsunami using the little script you wrote.

tsdl list.txt

Troubleshooting

If either server is behind a firewall, you'll need to setup port forwarding for UDP port 46224. Otherwise you'll experience that the servers can communicate, but not send data. (The source server will complain "no heartbeat since...")

After writing all this I realized a much easier method - the built-in get * feature works just fine.

On the source server, navigate to the directory where your files are and start the daemon with * passed in as your argument

cd /path/to/files
tsunamid *

Then on your destination server simply do

cd /path/to/files
tsunami connect your_source_ip_or_hostname_here get *