Performance Constrained by Low-Speed 300 KB/s Down, 39 KB/s Up ADSL Provisioning
Benchmarks conducted 18-19, 22-24, 26-28 February; 01-07, 15-22 March 2010
| BitTorrent Client | Time to Download ubuntu-9.10-desktop i386.iso |
Memory Use While Downloading | User CPU (MM:SS) | Sys CPU (MM:SS) | Engine | GUI |
|---|---|---|---|---|---|---|
| BitTornado 0.3.18 1st run | 53:33 | 19.3 MiB | 2:15 | 0:53 | Built-in | Curses |
| BitTornado 0.3.18 2nd run | 59:34 | 79.8 MiB | 7:51 | 3:47 | Built-in | Curses |
| BitTornado 0.3.18 3rd run | 59:05 | 90.3 MiB | 8:36 | 3:37 | Built-in | Curses |
| Deluge 1.2.0 1st run 1 | 57:23 | 49.2 MiB | 3:25 | 0:53 | rasterbar 0.14.8 | GTK+ |
| Deluge 1.2.1 1st run | 59:03 | 46.6 MiB | 3:47 | 1:00 | rasterbar 0.14.9 | GTK+ |
| Deluge 1.2.1 2nd run | 58:49 | 46.6 MiB | 3:24 | 0:58 | rasterbar 0.14.9 | GTK+ |
| Deluge 1.2900-dev 1st run 2 | 58:29 | 38.6 MiB | 3:20 | 0:54 | rasterbar 0.14.8 | GTK+ |
| Deluge 1.2900-dev 2nd run | 56:58 | 38.8 MiB | 3:27 | 0:55 | rasterbar 0.14.8 | GTK+ |
| Deluge 1.2900-dev 3rd run 3 | 57:00 | 38.3 MiB | 3:12 | 0:55 | rasterbar 0.15.0 | GTK+ |
| KTorrent 3.3.4 1st run | 56:16 | 19.9 MiB | N/A ^4 | N/A | Built-in | Qt |
| KTorrent 3.3.4 2nd run | 57:41 | 49.6 MiB | 2:28 | 1:50 | Built-in | Qt |
| KTorrent 3.3.4 3rd run | 58:47 | 55.5 MiB | 2:23 | 1:38 | Built-in | Qt |
| KTorrent 3.3.4 4th run | 57:57 | 51.9 MiB | 2:12 | 1:35 | Built-in | Qt |
| Mainline 4.4.0 1st run | 54:26 | 40.8 MiB | 3:18 | 1:02 | Built-in | Curses |
| Mainline 4.4.0 2nd run | 55:13 | 27.6 MiB | 3:39 | 1:17 | Built-in | Curses |
| Mainline 4.4.0 3rd run | 54:52 | 29.9 MiB | 3:51 | 1:28 | Built-in | Curses |
| Monsoon 0.21 1st run ^5 | 913:59 | 34.6 MiB | 158:57 | 19:18 | MonoTorrent 0.72 | GTK+ |
| Monsoon 0.21 2nd run | 75:39 | 30.4 MiB | 14:34 | 2:52 | MonoTorrent 0.72 | GTK+ |
| Monsoon 0.21 3rd run | 55:30 | 33.2 MiB | 11:05 | 2:31 | MonoTorrent 0.72 | GTK+ |
| Monsoon 0.21 4th run | 122.55 | 32.8 MiB | 24:06 | 4:22 | MonoTorrent 0.72 | GTK+ |
| qBittorrent 2.1.5 1st run | 55:26 | 33.2 MiB | 2:50 | 0:38 | rasterbar 0.14.8 | Qt |
| qBittorrent 2.1.5 2nd run | 55:23 | 36.9 MiB | 3:04 | 0:42 | rasterbar 0.14.8 | Qt |
| qBittorrent 2.1.6 1st run | 56:28 | 41.0 MiB | 3:07 | 0:44 | rasterbar 0.14.8 | Qt |
| qBittorrent 2.2.0 1st run | 56:28 | 36.4 MiB | 3:01 | 0:42 | rasterbar 0.14.9 | Qt |
| qBittorrent 2.2.0 2nd run | 55:15 | 34.6 MiB | 2:52 | 0:40 | rasterbar 0.14.9 | Qt |
| rTorrent 0.8.6 1st run 6 | 52:52 | invalid: 16.1 MiB | 0:16 | 0:19 | rakshasa 0.12.6 | Curses |
| rTorrent 0.8.6 2nd run 7 | 55:25 | invalid: 9.2 MiB | 0:47 | 0:22 | rakshasa 0.12.6 | Curses |
| rTorrent 0.8.6 3rd run | 53:53 | invalid: 5.8 MiB | 0:49 | 0:22 | rakshasa 0.12.6 | Curses |
| rTorrent 0.8.6 4th run | 53:44 | invalid: 4.4 MiB | 0:48 | 0:22 | rakshasa 0.12.6 | Curses |
| Transmission 1.76 1st run | 55:29 | 10.4 MiB | 1:30 | 0:31 | libtransmission | GTK+ |
| Transmission 1.90 1st run | 57:21 | 10.1 MiB | 1:33 | 0:45 | libtransmission | GTK+ |
| Transmission 1.91 1st run | 60:20 | N/A | 1:15 | 0:47 | libtransmission | GTK+ |
| Transmission 1.91 2nd run | 60:40 | 11.2 MiB | 1:15 | 0:45 | libtransmission | GTK+ |
| Transmission 1.91 3rd run | 59.46 | 10.6 MiB | 1:40 | 0:47 | libtransmission | GTK+ |
| Transmission 1.92 1st run | 54:54 | 10.3 MiB | 1:15 | 0:34 | libtransmission | GTK+ |
| Transmission 1.92 2nd run | 55:27 | 10.5 MiB | 1:11 | 0:34 | libtransmission | GTK+ |
| Transmission 1.92+ svn 10396 1st run | 54:32 | 11.7 MiB | 1:27 | 0:33 | libtransmission | GTK+ |
| Transmission 1.92+ svn 10396 2nd run | 54:35 | 10.8 MiB | 1:13 | 0:33 | libtransmission | GTK+ |
| uTorrent (Wine) 2.0 (18620) 1st run ^9 | 54:12 | 26.7 MiB | 2:06 | 1:37 | Built-in | Wine |
| uTorrent (Wine) 2.0 (18620) 2nd run | 53:52 | 20.8 MiB | 2:03 | 1:25 | Built-in | Wine |
| uTorrent (Wine) 2.0 (18620) 3rd run | 55:03 | 26.5 MiB | 2:12 | 1:44 | Built-in | Wine |
| Vuze 4.3.1.4 1st run | 53:40 | 246.5 MiB | 5:44 | 0:51 | Built-in | Java SWT |
| Vuze 4.3.1.4 Classic View 1st run | 53:58 | 243.5 MiB | 5:01 | 0:47 | Built-in | Java SWT |
| Vuze 4.3.1.4 Classic View 2nd run | 53:54 | 232.5 MiB | 4:54 | 0:48 | Built-in | Java SWT |
System: Asus A8V Deluxe mainboard, AMD Athlon(tm) 64 Processor 3200+. 2 GB DDR RAM
Network: 10/100 Ethernet, IPv4 NAT, ClarkConnect linux gateway/firewall/router PC ^10, AT&T branded 2Wire 2701HG-B router/modem set to Bridged Ethernet mode, i.e., only the modem component is active
Broadband: AT&T ADSL provisioned as 300/39 KB/s; 240/39 KB/s typically best actual
Operating System: Fedora 12 Linux - #1 SMP i686 athlon i386 GNU/Linux
Kernels 11: kernel-PAE-2.6.32.8-58.fc12.i686; kernel-PAE-2.6.32.9-67.fc12.i686; kernel-PAE-2.6.32.9-70.fc12.i686
Conditions:
-
Only one client was running per test. All tests were run on the same workstation, described above.
-
No other systems in the LAN were active on the Internet during tests, except for IRC chat (no DCC) by one person.
-
All tests were performed between 09:00 and 01:00 EST.
-
Each client was tested at least three times, never in back-to-back sequence, and usually on different days.
-
Each test was run under the unix time utility as: time [program-name] [options] ubuntu-9.10-desktop-i386.iso.torrent.
-
Each client was tested using its initial setup defaults, with two exceptions, made during a previous launch to edit preferences where possible, or incorporated into the command line when required:
(1) Each client was configured to use port 51421 (both TCP and UDP available); and
(2) Each client was configured to save torrents in a client-named directory on the ext4 filesystem used. -
Memory use ^12 was captured at 98% completion of each test run, by manually invoking: python ps_mem.py [client-name]
-
Immediately upon 100% completion and switch to seeding, each client was quit and its timed results recorded on exit.
-
Multiple client updates were released during the test span. I left all results in the table to show release variability in an active development environment.
These tests provide a decently rigorous picture of the capabilities and differences among several popular linux bittorrent clients, running on one system under one linux distribution over one type of ADSL broadband provisioning. Note that the tests are all for a single torrent, chosen for its large and well seeded swarm to provide as much reproducibility as any bittorrent swarm can likely offer. Among the things these tests do not provide are:\
(1) Scaling results for clients under multiple torrent loads;
(2) Performance differences between clients when each has been optimally configured beyond initial setup defaults;
(3) Performance differences between clients on more generously provisioned broadband;
(4) Features or performance of daemon, remote, web ui, or other variants on the standard client; and
(5) Effects (if any) of different linux distributions upon bittorrent client performance.
The single most significant performance constraint of these tests is the relatively low broadband speed as provisioned by the ISP. At such low speeds, choice of bittorrent client appears to make little difference in download metrics, leaving resource use and feature set as the primary salient factors. At this time, in this location, AT&T is not known to employ bittorrent throttling.
Thanks to Charles Kerr for posting an earlier version of these tests, upon which this version is based. Additional thanks to developers of various clients, whose help and suggestions improved these tests in many ways. All errors and inconsistencies in these tests are the fault of the author, Lacrocivious Acrophosist, who probably made the error specifically to annoy you, personally.
[*] Memory use reports for rTorrent by this python script are not representative of actual memory use by rTorrent while downloading. Developers involved with multiple competing clients tested here noted this discrepancy. One stated, "rTorrent relies more upon the (kernel) page cache table than most other clients, and so memory usage reported in the table does not necessarily reflect how much free memory a system would need to run rTorrent with satisfactory results."
During tests, rTorrent's info screen confirms its own reported memory use differs significantly from that reported by the python script. That screen's "Memory usage:" field varies dynamically during download between ~22.0 MB and ~40.0 MB, cycling in 30-second intervals, most often from a "floor" of ~25.0 MB and increasing to ~36.0 MB before falling back to ~25.0 MB and repeating. Thus, a rough estimate of actual memory use in these tests is ~38 MB.
[1]: Deluge 1.2.0 had an overdownloading bug frowned upon by some trackers; this issue is fixed in 1.2.1 and later versions. The key to this fix is libtorrent-rasterbar 0.14.9, more than changes to Deluge itself.
[2] Awaiting solution to a trivial 1.2.1 build problem on the test system, two interim runs used 1.2900-dev with libtorrent-rasterbar 0.14.8.
[3] Deluge recently switched from Subversion to Git; this build is from a fresh git pull of the master branch, incorporating libtorrent-rasterbar 0.15.0.
[4] KTorrent forks on load, preventing the time utility from
recording accurate start/end/CPU-use. Memory use difference on this run
may be due to this fork (the python ps_mem.py ktorrent report was
captured, but the time report was not). Subsequent tests were run
with time using the ktorrent --nofork option, which enables
time functionality.
[5] Monsoon has a poor endgame strategy. Repeatedly but not always, it would download to about 99% completion and then hang indefinitely on the last few pieces. Monsoon appears ill-suited for normal use, and was tested here only for comparison with prior tests.
[6] This test run used rTorrent from the Fedora 12 repository due to a naming conflict between libtorrent-rasterbar and libtorrent-rakshasa, before I found a workaround.
[7] This is the same libtorrent-rakshasa version as in the Fedora 12 repository, but both rTorrent and libtorrent-rakshasa were built from project source for this and subsequent test runs.
[8] PEPCAK attack. Terminal was in the wrong directory to find python ps_mem.py, and by the time I realized it the torrent had finished downloading.
[9] uTorrent memory use only. Wine components reported separate memory use: explorer.exe, 3.8 MiB; services.exe, 1.2 MiB; winedevice.exe, 2.8 MiB; and wineserver, 2.1 MiB; total 9.9 MiB. Wine version: wine-1.1.38-1.fc12.i686.
[10] ClarkConnect have rebranded themselves ClearOS. I have found their products exemplary since I began using them in 2003 (they don't pay me).
[11] Normal system updates, including the Fedora 12 updates-testing repository, were enabled during the test runs, and kernels changed during that span. Client tests were significantly staggered within the testing time span, and none of the operating system changes appeared to have any observable effect upon any client performance.
[12] I do not pretend to fully understand this python script. I am not a programmer. The script is well commented and used here for continuity with prior tests.
Is there a start-up time comparison of clients? with 10, 100 and 1000 torrents in its database perhaps