Jetty Netty benchmark

gistfile1.txt

-Xmx10g
m2.xlarge (4 virtual cores)

Both Jetty and Netty execute the same code--generate 8k of random bits and compute a sha1, returning it over the wire.

INTERNAL (Benchmark tool runs on same machine)
--------

Jetty:

ubuntu@ip-10-153-245-153:~/wrk$  wrk -t12 -c400 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    65.65ms  167.73ms   1.05s    92.71%
    Req/Sec     2.02k     1.33k   15.67k    77.43%
  657227 requests in 30.00s, 79.60MB read
  Socket errors: connect 0, read 0, write 0, timeout 35
Requests/sec:  21908.17
Transfer/sec:      2.65MB

ubuntu@ip-10-153-245-153:~$ wrk -t400 -c800 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
  400 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   791.67ms  348.56ms   1.52s    76.62%
    Req/Sec    59.10    267.40     7.44k    97.04%
  634569 requests in 29.95s, 76.86MB read
  Socket errors: connect 0, read 0, write 0, timeout 698
Requests/sec:  21189.80
Transfer/sec:      2.57MB

Netty:

ubuntu@ip-10-153-245-153:~$ wrk -t12 -c400 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.79ms    7.52ms  92.35ms   74.65%
    Req/Sec     1.94k   309.68     2.55k    86.12%
  672502 requests in 30.00s, 69.91MB read
Requests/sec:  22417.85
Transfer/sec:      2.33MB

ubuntu@ip-10-153-245-153:~$ wrk -t400 -c800 -d30s http://localhost:8080/
Running 30s test @ http://localhost:8080/
  400 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.19ms   15.02ms 146.59ms   58.61%
    Req/Sec    62.73     24.85   297.00     78.45%
  732090 requests in 29.86s, 76.10MB read
  Socket errors: connect 0, read 0, write 0, timeout 47
Requests/sec:  24513.85
Transfer/sec:      2.55MB



EXTERNAL (Benchmark tool runs from network)
--------

Jetty:

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  400 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    52.03ms  129.21ms   1.14s    96.15%
    Req/Sec    80.33     47.43   605.00     77.20%
  798971 requests in 29.96s, 96.77MB read
  Socket errors: connect 0, read 0, write 0, timeout 473
Requests/sec:  26671.30
Transfer/sec:      3.23MB

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  400 threads and 1600 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    91.24ms  131.17ms   1.04s    91.94%
    Req/Sec    79.43     56.79     1.28k    84.66%
  822948 requests in 29.96s, 99.67MB read
  Socket errors: connect 0, read 0, write 0, timeout 1320
Requests/sec:  27471.81
Transfer/sec:      3.33MB

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  5000 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   426.47ms  198.73ms   2.07s    67.76%
    Req/Sec     5.58      4.97   111.00     74.71%
  802760 requests in 27.94s, 97.23MB read
  Socket errors: connect 0, read 0, write 0, timeout 2566
Requests/sec:  28732.09
Transfer/sec:      3.48MB


Netty @m2.xlarge:

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  400 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.32ms   13.61ms 125.25ms   70.44%
    Req/Sec    76.55     26.27   377.00     73.38%
  876531 requests in 29.95s, 91.12MB read
  Socket errors: connect 0, read 0, write 0, timeout 149
Requests/sec:  29264.12
Transfer/sec:      3.04MB

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  400 threads and 1600 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    53.40ms   23.51ms 340.15ms   70.24%
    Req/Sec    77.69     22.22   349.00     78.20%
  877730 requests in 29.95s, 91.24MB read
  Socket errors: connect 0, read 0, write 0, timeout 601
Requests/sec:  29307.48
Transfer/sec:      3.05MB

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  5000 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   368.18ms  122.27ms   1.29s    69.19%
    Req/Sec     5.82      4.21    56.00     74.12%
  883874 requests in 27.61s, 91.88MB read
  Socket errors: connect 0, read 0, write 0, timeout 24
Requests/sec:  32018.62
Transfer/sec:      3.33MB

Running 30s test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  100 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.25ms   11.23ms  61.74ms   91.32%
    Req/Sec   301.84    161.60     1.00k    72.90%
  878652 requests in 29.99s, 91.34MB read
Requests/sec:  29302.65
Transfer/sec:      3.05MB



MASSIVE (A much longer sustained test with a large number of request threads and idle connections)
-------

Jetty:
26.5% RAM

Running 10m test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  6000 threads and 20000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.92s     2.88s   16.47s    88.40%
    Req/Sec     4.33      3.87   125.00     68.03%
  16539318 requests in 9.92m, 1.96GB read
  Socket errors: connect 0, read 0, write 121, timeout 40696
Requests/sec:  27782.09
Transfer/sec:      3.36MB

Netty
5.7% RAM

Running 10m test @ http://ec2-54-160-211-148.compute-1.amazonaws.com:8080/
  6000 threads and 20000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.08s     4.08s   31.35s    87.80%
    Req/Sec     4.41      3.07   105.00     75.95%
  17161794 requests in 9.92m, 1.74GB read
  Socket errors: connect 0, read 0, write 0, timeout 64255
Requests/sec:  28825.81
Transfer/sec:      3.00MB