Create a gist now

Instantly share code, notes, and snippets.

@methane /README.md
Last active Aug 29, 2015

What would you like to do?
Gazelle vs Meinheld

環境:

  • c3.8xlarge (32core)
  • Amazon Linux
  • Perl は yum install perl-devel したもの
  • Python は pyenv を利用して 2.7.9 をビルド
  • nginx は OpenResty (echo のため)

ボトルネックが変化してもなるべくベンチマークのオプションを変えずにCPUを使うため、 wrk, nginx, Webアプリそれぞれ16プロセス(orスレッド) を利用する。 ただし、 meinheld, minefield については、 16 プロセスだと thundering herd 問題を起こしているらしく性能が激しく劣化するので、 worker=8 にする。

Gazelle:

~/perl5/bin/start_server --path=/dev/shm/app.sock -- plackup  -s Gazelle --max-workers 16 --max-reqs-per-child 500000 -E prod -e 'sub{[200,["Content-Length", "12", "Content-Type", "text/plain"],["hello world\n"]]}'
start_server (pid:35909) starting now...
starting new worker 35910

curl:

$ curl -v http://localhost:8000/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:8000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server openresty/1.7.4.1 is not blacklisted
< Server: openresty/1.7.4.1
< Date: Sun, 21 Dec 2014 15:53:26 GMT
< Content-Type: text/plain
< Content-Length: 12
< Connection: keep-alive
<
hello world
* Connection #0 to host localhost left intact

wrk:

$ ./wrk -c128 -t16 -d20 http://localhost:8000/
Running 20s test @ http://localhost:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   696.85us  834.34us  20.43ms   90.60%
    Req/Sec    13.98k     4.37k   28.22k    69.69%
  4137566 requests in 20.00s, 650.87MB read
Requests/sec: 206899.59
Transfer/sec:     32.55MB

$ ./wrk -c128 -t16 -d20 http://localhost:8000/
Running 20s test @ http://localhost:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   663.28us  773.26us  21.57ms   92.98%
    Req/Sec    14.00k     4.05k   26.89k    65.85%
  4142212 requests in 20.00s, 651.61MB read
Requests/sec: 207129.00
Transfer/sec:     32.58MB

$ ./wrk -c128 -t16 -d20 http://localhost:8000/
Running 20s test @ http://localhost:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   637.86us  762.55us  15.69ms   93.94%
    Req/Sec    14.16k     3.76k   27.56k    66.39%
  4200235 requests in 20.00s, 660.73MB read
Requests/sec: 210032.07
Transfer/sec:     33.04MB

$ ./wrk -c128 -t16 -d20 http://localhost:8000/
Running 20s test @ http://localhost:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   655.42us  776.66us  21.19ms   93.61%
    Req/Sec    14.01k     3.95k   26.22k    66.76%
  4146750 requests in 20.00s, 652.32MB read
Requests/sec: 207358.57
Transfer/sec:     32.62MB
res = b'Hello, World\n'
res_len = str(len(res))
def app(env, start):
start("200 OK", [('Content-Type', 'text/plain'), ('Content-Length', res_len)])
return [res]

curl

wrk$ curl -v http://localhost:8000/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:8000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server openresty/1.7.4.1 is not blacklisted
< Server: openresty/1.7.4.1
< Date: Sun, 21 Dec 2014 16:05:24 GMT
< Content-Type: text/plain
< Content-Length: 13
< Connection: keep-alive
<
Hello, World
* Connection #0 to host localhost left intact

wrk

bench$ gunicorn --keep-alive=120 -b unix:/dev/shm/app.sock -k meinheld.gmeinheld.MeinheldWorker -w 8 hello:app
wrk$ ./bench.sh
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.91ms    2.07ms 119.74ms   95.76%
    Req/Sec    12.99k     4.14k   42.89k    84.32%
  3891565 requests in 20.00s, 615.89MB read
Requests/sec: 194597.52
Transfer/sec:     30.80MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.26ms    1.66ms  39.13ms   86.43%
    Req/Sec    11.61k     5.36k   23.06k    71.80%
  3527117 requests in 20.00s, 558.21MB read
Requests/sec: 176373.65
Transfer/sec:     27.91MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.37ms    1.83ms 100.25ms   85.37%
    Req/Sec    11.06k     5.35k   21.25k    70.19%
  3358701 requests in 20.00s, 531.56MB read
Requests/sec: 167951.54
Transfer/sec:     26.58MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.40ms    1.86ms  93.97ms   84.13%
    Req/Sec    11.13k     5.43k   21.29k    69.58%
  3375888 requests in 20.00s, 534.28MB read
Requests/sec: 168804.15
Transfer/sec:     26.72MB

keep-alive

bench$ gunicorn --keep-alive=120 -b unix:/dev/shm/app.sock -k meinheld.gmeinheld.MeinheldWorker -w 16 hello:app
wrk$ ./bench.sh
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   540.14us    1.33ms  21.50ms   97.34%
    Req/Sec    20.01k     3.83k   39.92k    83.01%
  6125419 requests in 20.00s, 0.95GB read
Requests/sec: 306301.73
Transfer/sec:     48.48MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   555.74us    1.01ms  33.29ms   94.99%
    Req/Sec    19.38k     5.32k   48.77k    77.78%
  5823292 requests in 20.00s, 0.90GB read
Requests/sec: 291193.69
Transfer/sec:     46.08MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   563.98us    1.20ms  23.43ms   96.01%
    Req/Sec    19.61k     5.24k   47.67k    79.26%
  5885924 requests in 20.00s, 0.91GB read
Requests/sec: 294325.97
Transfer/sec:     46.58MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   504.93us    0.93ms  16.98ms   96.13%
    Req/Sec    19.87k     4.76k   49.15k    79.37%
  5981555 requests in 20.00s, 0.92GB read
Requests/sec: 299105.97
Transfer/sec:     47.34MB

keep-alive + direct

bench$ gunicorn --keep-alive=120 -b :8000 -k meinheld.gmeinheld.MeinheldWorker -w 16 hello:app
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   272.53us  306.04us  16.59ms   95.69%
    Req/Sec    31.26k     8.75k   63.44k    66.10%
  9403925 requests in 20.00s, 1.43GB read
Requests/sec: 470287.53
Transfer/sec:     73.11MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   258.08us  241.60us  16.66ms   94.22%
    Req/Sec    32.10k     7.26k   60.33k    63.79%
  9650665 requests in 20.00s, 1.47GB read
Requests/sec: 482590.29
Transfer/sec:     75.02MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   265.34us  345.25us  28.04ms   97.20%
    Req/Sec    31.57k     6.81k   63.11k    68.04%
  9502529 requests in 20.00s, 1.44GB read
Requests/sec: 475175.82
Transfer/sec:     73.87MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   269.89us  286.67us  16.40ms   95.73%
    Req/Sec    30.98k     7.78k   61.89k    69.86%
  9328887 requests in 19.99s, 1.42GB read
Requests/sec: 466600.05
Transfer/sec:     72.53MB

curl

wrk$ curl -v http://localhost:8000/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:8000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server openresty/1.7.4.1 is not blacklisted
< Server: openresty/1.7.4.1
< Date: Sun, 21 Dec 2014 17:05:12 GMT
< Content-Type: text/plain
< Content-Length: 13
< Connection: keep-alive
<
Hello, World
* Connection #0 to host localhost left intact

wrk

bench$ gunicorn --keep-alive=120 -b unix:/dev/shm/app.sock -k minefield.gminefield.MinefieldWorker -w 8 hello:app
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.39ms    2.15ms  66.41ms   87.59%
    Req/Sec    12.70k     6.66k   23.75k    67.40%
  3847773 requests in 20.00s, 608.96MB read
Requests/sec: 192408.12
Transfer/sec:     30.45MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.60ms    2.44ms 143.21ms   86.03%
    Req/Sec    12.51k     6.80k   24.35k    68.16%
  3797459 requests in 20.00s, 600.99MB read
Requests/sec: 189890.87
Transfer/sec:     30.05MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.69ms    2.27ms  80.73ms   86.65%
    Req/Sec    11.28k     7.02k   23.47k    54.93%
  3431201 requests in 20.00s, 543.03MB read
Requests/sec: 171572.33
Transfer/sec:     27.15MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.59ms    2.15ms 176.17ms   84.87%
    Req/Sec    11.68k     7.01k   24.06k    58.03%
  3542297 requests in 20.00s, 560.61MB read
Requests/sec: 177130.41
Transfer/sec:     28.03MB

keep-alive

bench$ gunicorn --keep-alive=120 -b unix:/dev/shm/app.sock -k minefield.gminefield.MinefieldWorker -w 16 hello:app
wrk$ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   511.02us    1.12ms  30.59ms   96.46%
    Req/Sec    20.77k     5.00k   30.80k    81.01%
  6345215 requests in 20.00s, 0.98GB read
Requests/sec: 317293.83
Transfer/sec:     50.22MB

wrk$ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   373.41us  602.57us  28.76ms   97.03%
    Req/Sec    21.99k     3.92k   56.11k    74.47%
  6656703 requests in 20.00s, 1.03GB read
Requests/sec: 332864.76
Transfer/sec:     52.68MB

wrk$ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   366.64us  503.52us  25.25ms   96.16%
    Req/Sec    22.48k     4.01k   35.89k    71.93%
  6717099 requests in 20.00s, 1.04GB read
Requests/sec: 335890.10
Transfer/sec:     53.16MB

wrk$ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   644.40us    1.14ms  19.23ms   93.10%
    Req/Sec    19.34k     6.60k   34.38k    76.34%
  5838727 requests in 20.00s, 0.90GB read
Requests/sec: 291965.50
Transfer/sec:     46.21MB

keep-alive + direct

wrk$ ./bench.sh
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   225.42us  245.19us  20.29ms   96.89%
    Req/Sec    36.37k     7.59k   69.33k    65.01%
  10931314 requests in 20.00s, 1.67GB read
Requests/sec: 546636.22
Transfer/sec:     85.50MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   219.49us  243.48us  28.45ms   96.81%
    Req/Sec    37.41k     7.34k   62.44k    66.28%
  11226829 requests in 20.00s, 1.71GB read
Requests/sec: 561405.87
Transfer/sec:     87.81MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   218.39us  253.50us  16.72ms   97.24%
    Req/Sec    37.55k     7.43k   62.78k    65.02%
  11288116 requests in 20.00s, 1.72GB read
Requests/sec: 564465.83
Transfer/sec:     88.28MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   219.41us  288.07us  24.40ms   97.70%
    Req/Sec    37.86k     8.17k   70.00k    65.67%
  11357831 requests in 20.00s, 1.73GB read
Requests/sec: 567968.45
Transfer/sec:     88.83MB

curl:

$ curl -v http://localhost:8000/hello
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /hello HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:8000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server openresty/1.7.4.1 is not blacklisted
< Server: openresty/1.7.4.1
< Date: Sun, 21 Dec 2014 15:37:43 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
<
Hello, World
* Connection #0 to host localhost left intact

wrk:

$ ./wrk -c128 -t16 -d20 http://localhost:8000/hello
Running 20s test @ http://localhost:8000/hello
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   156.02us  179.21us  22.56ms   95.23%
    Req/Sec    50.04k     8.03k   74.89k    79.67%
  14807446 requests in 20.00s, 2.54GB read
Requests/sec: 740445.97
Transfer/sec:    129.90MB
worker_processes 16;
daemon off;
events {
worker_connections 10000;
}
http {
include mime.types;
access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
etag off;
upstream app {
server unix:/dev/shm/app.sock;
#keepalive 16; # keep-alive: on の場合
}
server {
location / {
proxy_pass http://app;
# keep-alive: on の場合
#proxy_http_version 1.1;
#proxy_set_header Connection "";
}
location = /hello {
echo 'Hello, World';
}
}
}

config

uwsgi:

bench$ uwsgi -s /dev/shm/app.sock --master --workers=8 --wsgi=hello:app --disable-logging

nginx:

-            proxy_pass http://app;
+            include uwsgi_params;
+            uwsgi_pass app;

wrk

wrk$ ./bench.sh
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   712.07us    1.04ms  19.76ms   94.37%
    Req/Sec    15.31k     4.09k   23.67k    83.78%
  4624729 requests in 20.00s, 732.59MB read
  Non-2xx or 3xx responses: 4017
Requests/sec: 231258.49
Transfer/sec:     36.63MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.86ms    1.29ms  17.02ms   92.37%
    Req/Sec    14.43k     4.84k   24.00k    78.61%
  4377470 requests in 20.00s, 693.09MB read
  Non-2xx or 3xx responses: 1826
Requests/sec: 218893.41
Transfer/sec:     34.66MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   748.33us    1.31ms  16.64ms   95.32%
    Req/Sec    15.24k     4.04k   24.32k    84.18%
  4637451 requests in 20.00s, 734.21MB read
  Non-2xx or 3xx responses: 1693
Requests/sec: 231899.90
Transfer/sec:     36.72MB
+ ./wrk -c128 -t16 -d20 http://127.0.0.1:8000/
Running 20s test @ http://127.0.0.1:8000/
  16 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   801.55us    1.17ms  20.03ms   92.64%
    Req/Sec    14.64k     4.67k   26.76k    80.87%
  4433503 requests in 20.00s, 701.95MB read
  Non-2xx or 3xx responses: 1741
Requests/sec: 221696.17
Transfer/sec:     35.10MB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment