gistfile1.txt

Hello!

After series of experiments with ftp clients tuning for both uploading
and downloading with NCBI's ftp server we got to the conclusion that the
configuration of ftp client seriously affects the speed you get.

We recommend to increase the buffer size to 32MB, it helps for both
uploads and downloads.

In ncftp command line utility the buffer size can be changed via "set
so-bufsize" command.

Example, download from st-va colo to be-md:

ncftp / > set so-bufsize 0

ncftp / > get -z large /dev/null
/dev/null:                      1.00 GB   21.27 MB/s

ncftp / > set so-bufsize 33554432

ncftp / > get -z large /dev/null
/dev/null:                      1.00 GB   52.70 MB/s



For lukemftp (default ftp client for SUSE linux and some
other linux flavors) the change of buffer size is done
via commands:
	sndbuf 33554432
	rcvbuf 33554432


Example, uploading from NCBI to ftp.YYYYY.mit.edu:

ftp> put 100MB 100MB-1
local: 100MB remote: 100MB-1
229 Entering Extended Passive Mode (|||60355|)
150 Opening BINARY mode data connection for 100MB-1
100% |**********************|   100 MB    1.53 MB/s    00:00 ETA
226 Transfer complete.
104857600 bytes sent in 01:05 (1.52 MB/s)
ftp> sndbuf 33554432
Socket buffer sizes: send 33554432, receive 87380.
ftp> rcvbuf 33554432
Socket buffer sizes: send 33554432, receive 33554432.
ftp> put 100MB 100MB-2
local: 100MB remote: 100MB-2
229 Entering Extended Passive Mode (|||60469|)
150 Opening BINARY mode data connection for 100MB-2
100% |**********************|   100 MB    4.52 MB/s    00:00 ETA
226 Transfer complete.
104857600 bytes sent in 00:23 (4.33 MB/s)

Further reading: "Guide to Bulk Data Transfer over a WAN"
	http://fasterdata.es.net/

We have good experience with lftp on unix: http://lftp.yar.ru/
and Filezilla on Windows: http://filezilla.sourceforge.net/

Dima

beloslyu At ncbi dot nlm dot nih dot gov