HttpResults Round2.md

UPDATE:

These results are invalid. Some of the server implementations don't parse correctly and rust-tokio/ponylang-tcp don't seem to parse at all. See here for better benchmarks: https://gist.github.com/kprotty/5a41e9612657de00788478a7dde43d78

====

wrk -t4 -c128 -d10 --latency http://localhost:12345

• Machine:

  • Intel Core i7-6700k (4 cores, 8 threads, 4.2ghz)
  • 16GB DDR4 2400mhz RAM
  • Arch Linux, Kernel 5.2.8

• C non-blocking (https://pastebin.com/VHMmjFhy)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.03us   66.97us   6.76ms   99.96%
    Req/Sec   456.66k    49.47k  705.49k    88.00%
  Latency Distribution
     50%    5.00us
     75%    5.00us
     90%    6.00us
     99%    9.00us
  4533700 requests in 10.06s, 216.18MB read
Requests/sec: 450520.28
Transfer/sec:     21.48MB

• C multi-threaded (https://pastebin.com/B6LGaTRq)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    71.70ms   53.40ms 320.61ms   62.03%
    Req/Sec   794.31k   172.83k    1.93M    76.13%
  Latency Distribution
     50%   63.31ms
     75%  109.61ms
     90%  151.26ms
     99%  196.52ms
  31478929 requests in 10.07s, 1.47GB read
Requests/sec: 3126434.62
Transfer/sec:    149.08MB

• C non-blocking multi-threaded (EPOLLET) (https://pastebin.com/N55mku4G)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.77ms   20.30ms 384.52ms   98.24%
    Req/Sec     0.99M   726.03k    4.18M    84.12%
  Latency Distribution
     50%   36.00us
     75%    4.38ms
     90%   12.04ms
     99%   39.34ms
  29999649 requests in 10.09s, 1.40GB read
  Socket errors: connect 0, read 0, write 2, timeout 3
Requests/sec: 2971965.71
Transfer/sec:    141.71MB

• C non-blocking multi-threaded (EPOLLONESHOT) (https://pastebin.com/3jDZp8DC)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    14.63ms  102.86ms   1.38s    98.44%
    Req/Sec     0.93M   418.25k    3.91M    82.83%
  Latency Distribution
     50%   22.00us
     75%    3.73ms
     90%   12.33ms
     99%  567.99ms
  28006935 requests in 10.09s, 1.30GB read
  Socket errors: connect 0, read 4, write 2, timeout 4
Requests/sec: 2775978.13
Transfer/sec:    132.37MB

• Rust actix (https://pastebin.com/yTN4mp5c)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   719.41us  181.72us   9.35ms   94.70%
    Req/Sec    44.72k     4.73k   58.48k    68.75%
  Latency Distribution
     50%  684.00us
     75%  792.00us
     90%  825.00us
     99%    1.35ms
  1779562 requests in 10.05s, 84.86MB read
Requests/sec: 177030.63
Transfer/sec:      8.44MB

• Rust actix-web (https://pastebin.com/p4GxGkJJ)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.09ms    2.58ms  30.16ms   91.00%
    Req/Sec   118.84k    15.96k  140.87k    68.75%
  Latency Distribution
     50%  141.00us
     75%  596.00us
     90%    3.19ms
     99%   13.36ms
  4730269 requests in 10.04s, 577.43MB read
Requests/sec: 471160.26
Transfer/sec:     57.51MB

• Rust tokio (https://pastebin.com/byUBd8Wy)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    78.26ms   54.56ms 297.56ms   66.19%
    Req/Sec     7.90M   219.58k    8.25M    90.77%
  Latency Distribution
     50%   70.14ms
     75%  112.62ms
     90%  156.40ms
     99%  232.40ms
  316801024 requests in 10.19s, 14.75GB read
Requests/sec: 31080970.08
Transfer/sec:      1.45GB

• Golang evio (https://pastebin.com/P04PEwYR)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.44ms    3.45ms  48.21ms   90.34%
    Req/Sec   135.90k     5.86k  147.08k    77.00%
  Latency Distribution
     50%  159.00us
     75%  557.00us
     90%    4.72ms
     99%   16.72ms
  5408490 requests in 10.03s, 257.90MB read
Requests/sec: 538966.01
Transfer/sec:     25.70MB

• Elixir TCP (https://pastebin.com/eWSqBM5a)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    19.98ms  134.24ms   1.65s    97.35%
    Req/Sec   146.14k    38.66k  243.66k    61.75%
  Latency Distribution
     50%  263.00us
     75%  399.00us
     90%    1.91ms
     99%  779.09ms
  5823156 requests in 10.06s, 277.67MB read
  Socket errors: connect 0, read 0, write 0, timeout 12
Requests/sec: 579064.50
Transfer/sec:     27.61MB

• Ponylang TCP (https://pastebin.com/YCYQKtPM)

Running 10s test @ http://localhost:12345
  4 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    40.45ms   28.17ms 165.03ms   65.67%
    Req/Sec     7.78M   186.71k    8.19M    89.64%
  Latency Distribution
     50%   36.16ms
     75%   58.49ms
     90%   81.68ms
     99%  118.68ms
  309517082 requests in 10.07s, 14.41GB read
Requests/sec: 30732328.55
Transfer/sec:      1.43GB

NewResults.md

wrk -t6 -c128 -d10 --latency http://localhost:12345 System: Arch Linux 5.4, Ryzen 5 2600, 16gb 3000mhz ddr4 ram

• Zig (zhp) (https://pastebin.com/EbN2gYWa)

Running 10s test @ http://localhost:12345
  6 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   487.16us  837.43us  39.52ms   94.52%
    Req/Sec    59.49k    14.27k   86.72k    56.67%
  Latency Distribution
     50%  386.00us
     75%  474.00us
     90%  566.00us
     99%    4.71ms
  3552616 requests in 10.08s, 477.71MB read
Requests/sec: 352397.66
Transfer/sec:     47.39MB

• Golang (fasthttp) (https://pastebin.com/16dmjLCu)

Running 10s test @ http://localhost:12345
  6 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   494.83us    1.12ms  24.03ms   93.83%
    Req/Sec    70.68k     9.40k   92.71k    66.44%
  Latency Distribution
     50%  221.00us
     75%  359.00us
     90%  773.00us
     99%    5.92ms
  4222556 requests in 10.08s, 583.91MB read
Requests/sec: 418838.30
Transfer/sec:     57.92MB

@RUSshy yea but these tests are more io bound than cpu bound and there wasnt really much cpu work being done. All of them also werent written correctly

@kprotty, Can you run this https://github.com/frmdstryr/zhp/blob/master/tests/raw.zig and see how it does on your system? I'm curious how tokio seems to do an order of magnitude faster than the others... (you might have to apply this patch to zig ziglang/zig@5279e49)

@frmdstryr its because its cheating by not actually parsing data but instead just sending as many bytes as possible. I believe all the results are invalid by now due to each having their own wierd quirks

Zig with ZHP (release-fast=true) (https://pastebin.com/8L4WZpbE):

wrk -t6 -c128 -d10 --latency http://localhost:9000
Running 10s test @ http://localhost:9000
  6 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.36ms    1.69ms  77.39ms   96.67%
    Req/Sec    15.37k     3.56k   26.74k    64.67%
  Latency Distribution
     50%    1.14ms
     75%    1.44ms
     90%    2.12ms
     99%    4.06ms
  919039 requests in 10.02s, 112.19MB read
Requests/sec:  91705.14
Transfer/sec:     11.19MB

Go with fasthttp (https://pastebin.com/RAWajkmH)

wrk -t6 -c128 -d10 --latency http://localhost:12345
Running 10s test @ http://localhost:12345
  6 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.51ms   27.02ms 319.23ms   89.30%
    Req/Sec    17.95k    11.50k   60.16k    71.04%
  Latency Distribution
     50%  720.00us
     75%    4.30ms
     90%   42.35ms
     99%  121.67ms
  1031267 requests in 10.10s, 142.61MB read
Requests/sec: 102112.70
Transfer/sec:     14.12MB

on a 2017 macbook pro

lol one shouldn't do benchmarks on laptop especially MacBook.

Tokio one looks outlier but its not doing any hard work like parsing etc.

why would that matter im not testing their capabilities im comparing how they do on equal conditions :3

@shirshak55 @courtier These results are invalid due as many of the implementations here don't even parse correctly. See this instead: https://gist.github.com/kprotty/5a41e9612657de00788478a7dde43d78

invalid how man im benchmarking two packages against each other... i dont care about implementations i just wanna compare the speeds of two packages

@courtier I explained how above with an edit. Implementation affects things like your generic "speed" metric... If you don't know that two packages are achieving the same thing, then how are any results you get to compare them useful?

superficially they are doing the same thing and thats what i want, compare as superficially as possible

If one library is dropping requests at random due to implementation bugs, they're no longer doing the same thing and comparing superficially still isn't useful as the wrk results would just show undefined variance in throughput, depending on which requests were dropped during that run.

@kprotty have you seen https://www.techempower.com/benchmarks/ It'd be nice to get a zig project on there...

I added zhp here but it is showing a lot of read errors at the highest currency level. The drogon framework is about twice as fast...