gistfile1.md

Nginx vs h2o (http/1.1)

Environment

Web Server

AWS c3.xlarge on VPC (ap-northeast-1)

LoadWorker

AWS c3.8xlarge on VPC (ap-northeast-1)

Nginx

$ nginx -V
nginx version: nginx/1.6.2
built by gcc 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC)

h2o

• libyaml enable
• libuv 1.0.0 disable
• wslay disable

d8a901036b9bd5d7a4b334791cda06175330b39b

Tunning

Kernel

$ sudo sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

$ cat /etc/security/limits.d/ec2-user.conf
ec2-user soft nproc 655350
ec2-user hard nproc 655350
ec2-user soft nofile 655350
ec2-user hard nofile 655350

Nginx

$ sudo cat /etc/nginx/nginx.conf
user  ec2-user;
worker_processes  4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
events {
    worker_connections 20480;
    multi_accept on;
    use epoll;
}

http {
    default_type  application/octet-stream;

    access_log  off;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;
    server_tokens   off;
    index   index.html;

    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 8m;
    large_client_header_buffers 2 1k;

    client_body_timeout 12;
    client_header_timeout 12;

    keepalive_timeout 65;
    keepalive_requests 1000000;
    send_timeout 10;

    open_file_cache max=100000 inactive=20s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    server {
        listen       80;
        server_name  localhost;
        root         /home/ec2-user/h2o/examples/doc_root;
    }
}

h2o

$ cat /home/ec2-user/h2o/examples/h2o/h2o_bench.conf
# to find out the configuration commands, run: h2o --help
max-connections: 20480
num-threads: 4
listen: 80
hosts:
  default:
    paths:
      /:
        file.dir: /home/ec2-user/h2o/examples/doc_root

test file

$ ls -l /home/ec2-user/h2o/examples/doc_root/1
-rw-rw-r-- 1 ec2-user ec2-user 1 11月 12 02:48 /home/ec2-user/h2o/examples/doc_root/1

Result

Nginx

[ec2-user@ip-172-31-12-206 wrk]$ ./wrk -t 2 -c 10000 -d 10 http://172.31.27.125/1
Running 10s test @ http://172.31.27.125/1
  2 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    79.34ms  111.40ms   2.09s    87.18%
    Req/Sec    37.19k    14.02k   56.71k    69.89%
  721931 requests in 10.00s, 167.30MB read
  Socket errors: connect 0, read 0, write 0, timeout 16410
Requests/sec:  72162.29
Transfer/sec:     16.72MB

[ec2-user@ip-172-31-27-125 ~]$ cat /tmp/strace_nginx.out
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 30.34    0.003355           0     73036           setsockopt
 25.88    0.002862           0     64293     27407 recvfrom
 15.23    0.001684           0     36518           sendfile
 14.71    0.001627           0     36518           writev
 11.91    0.001317          14        93           epoll_wait
  1.16    0.000128           0       717           close
  0.58    0.000064           0       725         1 accept4
  0.20    0.000022           0        94           gettimeofday
  0.00    0.000000           0         1           stat
  0.00    0.000000           0         4           brk
  0.00    0.000000           0       734           epoll_ctl
------ ----------- ----------- --------- --------- ----------------
100.00    0.011059                212733     27408 total

h2o

[ec2-user@ip-172-31-12-206 wrk]$ ./wrk -t 2 -c 10000 -d 10 http://172.31.27.125:8888/1
Running 10s test @ http://172.31.27.125:8888/1
  2 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   123.95ms  178.87ms   8.21s    98.66%
    Req/Sec     4.59k   115.78     4.83k    64.86%
  90775 requests in 10.00s, 19.44MB read
  Socket errors: connect 0, read 874, write 0, timeout 31151
Requests/sec:   9075.48
Transfer/sec:      1.94MB

[ec2-user@ip-172-31-27-125 ~]$ cat /tmp/strace_h2o.out
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 30.00    0.076597           0    278881     92082 read
 21.14    0.053979          95       569           select
 17.77    0.045380           0     92855           writev
 10.93    0.027912           0     92855       884 open
  9.50    0.024250           0     94068           close
  8.65    0.022090           0     91971           fstat
  0.73    0.001864           1      2497       400 accept4
  0.49    0.001260           1      1599           gettimeofday
  0.44    0.001119           1      2097           setsockopt
  0.27    0.000691           0      2097           fcntl
  0.06    0.000159           0       402        94 futex
  0.01    0.000027           1        25           madvise
------ ----------- ----------- --------- --------- ----------------
100.00    0.255328                659916     93460 total