Benchmarking Hops with various configurations (spoiler: Mongrel2 + LuaJIT 2 == FAST!)

hops_lighttpd_fcgi_lua51_maxprocs1.txt

$ ab -n 1000 -c 50 http://10.1.1.82:80/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        lighttpd/1.4.28
Server Hostname:        10.1.1.82
Server Port:            80

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.488 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      418418 bytes
HTML transferred:       257257 bytes
Requests per second:    2048.74 [#/sec] (mean)
Time per request:       24.405 [ms] (mean)
Time per request:       0.488 [ms] (mean, across all concurrent requests)
Transfer rate:          837.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       4
Processing:     2   23   2.7     23      27
Waiting:        2   23   2.7     23      27
Total:          7   24   2.2     24      30
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     24
  66%     24
  75%     25
  80%     25
  90%     25
  95%     26
  98%     26
  99%     26
 100%     30 (longest request)

hops_lighttpd_fcgi_luajit2_maxprocs1.txt

$ ab -n 1000 -c 50 http://10.1.1.82:80/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        lighttpd/1.4.28
Server Hostname:        10.1.1.82
Server Port:            80

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.371 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      418418 bytes
HTML transferred:       257257 bytes
Requests per second:    2691.84 [#/sec] (mean)
Time per request:       18.575 [ms] (mean)
Time per request:       0.371 [ms] (mean, across all concurrent requests)
Transfer rate:          1099.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.9      1       5
Processing:     4   17   1.9     18      20
Waiting:        4   17   1.9     18      20
Total:         10   18   1.3     18      23

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     18
  75%     19
  80%     19
  90%     19
  95%     20
  98%     20
  99%     22
 100%     23 (longest request)

hops_xavante_lua51.txt

$ ab -n 1000 -c 50 http://10.1.1.82:6789/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Xavante
Server Hostname:        10.1.1.82
Server Port:            6789

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.699 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      397000 bytes
HTML transferred:       257000 bytes
Requests per second:    1431.30 [#/sec] (mean)
Time per request:       34.933 [ms] (mean)
Time per request:       0.699 [ms] (mean, across all concurrent requests)
Transfer rate:          554.91 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       5
Processing:     3   21  17.2     18     224
Waiting:        3   20  17.3     18     224
Total:          7   22  17.2     19     225
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     20
  75%     20
  80%     21
  90%     22
  95%     29
  98%     63
  99%     63
 100%    225 (longest request)

hops_xavante_luajit2.txt

$ ab -n 1000 -c 50 http://10.1.1.82:6789/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Xavante
Server Hostname:        10.1.1.82
Server Port:            6789

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.383 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      397000 bytes
HTML transferred:       257000 bytes
Requests per second:    2613.76 [#/sec] (mean)
Time per request:       19.130 [ms] (mean)
Time per request:       0.383 [ms] (mean, across all concurrent requests)
Transfer rate:          1013.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       4
Processing:     2   13  11.3     12     216
Waiting:        2   12  11.3     12     216
Total:          6   14  11.5     13     219
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     13
  66%     13
  75%     14
  80%     14
  90%     14
  95%     17
  98%     23
  99%     24
 100%    219 (longest request)

lighttpd.conf

# This is the most basic lighttpd.conf (derived from the default one shipped with
# the Debian package of Lighttpd) that can be used to test Hops with FastCGI.

server.modules = (
    "mod_access",
    "mod_fastcgi",
    "mod_rewrite",
)

# Replace with the full path to the 'example' directory of Hops.
server.document-root  =  "/home/adaniele/development/hops/example"
server.errorlog       = "/var/log/lighttpd/error.log"
server.pid-file       = "/var/run/lighttpd.pid"
server.username       = "www-data"
server.groupname      = "www-data"
url.access-deny       = ( "~", ".inc" )
static-file.exclude-extensions = ( ".fcgi" )

fastcgi.server = (
    ".lua" => (
        "localhost" => (
            "socket"   => "/tmp/lua-fastcgi_hops.socket",

            # Replace with the actual path on your system where the
            # wsapi.fcgi file (provided by wsapi-fcgi) is located.
            # NOTE: to switch between Lua 5.1 and Luajit2 you can edit
            # wsapi.fcgi so that it points to a different executable
            # used by the 'exec' command.
            "bin-path" => "/home/adaniele/.luarocks/bin/wsapi.fcgi",

            # Specify how many Lua processes are used by FastCGI
            # to elaborate requests. Setting max-procs to 1 means
            # using just one Lua process.
            "min-procs" => 1,
            "max-procs" => 1,
        )
    )
)

url.rewrite-once = (
    "^/([^.]+)?$" => "/app.lua/$1",
)

Thanks for the benchmarks and info - great to see you guys playing around with my code. :) BTW I'm pretty sure Orbit, Mercury and other web frameworks would have similar performance though too, there's nothing too special about Hops in terms of what it does to be fast. Really the credit goes entirely to Lua/LuaJIT and WSAPI.

Well yeah, it actually turned into a quick and nice demonstration about how the Mongrel2+Luajit2 pair can be really fast allowing for an high number of RPS with a very low memory footprint and CPU usage. Sticking with Hops as a reference microframework for these tests is now a matter of consistency, maybe I'll do the same with other frameworks such as Orbit just out of curiosity :-) Anyway, nice work on Hops!

Hi Danielle,
this all kicks ass ... where is it heading to, are you gonna make something other people can easily re-use .. I ask because I am becoming more and more of a Lua-junky.

Some questions:
1.) why not run this on bare metal?
2.) use 4 cores, or 6 cores - get some BIG benchmark numbers (like 30-50K)
3.) please please plug this all into a non-blocking redis backend framework ... please please :)

I hope all of this stuff results in some sort of real framework, something I can just be lazy and use, and all this work, you should write it up in a blog, cause Lua needs the press, its the absolute shit and people just dont know about it.

nice work ... keep going

• jak

Thanks! Glad you're liking Hops. I'm currently working on better template handling (caching, responding with different templates based on content type headers, the request and extension like Rails), but other than that the focus is more on ease of use than high performance. A certain good level of performance is guaranteed just by using Lua and WSAPI.

If you're looking for a non-blocking web development API for Lua then you might want to check out LuaNode.

@JakSprats: hey Jak it's been a while, and it's good to know that you are enjoying Lua more and more. In reply to your questions:

1. the reason is simple: we, at work, rely heavily on our in-house virtualization infrastructure. We still have a few bare-metal installations, but that hardware is relatively old.
2. I can still configure my virtual machine to use 8 cores :-) That ESX host is brand new and still unused aside from some test stuff, so there's actually nothing running while I do the tests.
3. I don't know how and if a WSAPI-based application can behave with non-blocking stuff, but aside from that I'd love to make redis-lua completely async with the next major version. As @norman pointed out you should definitely take a look at LuaNode.

@norman: Jak and performances go hand in hand ;-)

@nrk hey Danielle, 1 & 2 were just me wanting to see higher numbers, I can do the math in my head already, and this platform would smoke on a 6 core @3.2Ghz :)
I asked Ignacio of LuaNode about 3 months ago about redis integration and he said: no time right now.
So the non-blocking Lua-webserver w/ nonblocking redis backend remains to-be-done (redis2-nginx-module has done it, but I remember you wanted it done differently than they did it).
Anyways, I am polluting this gist, I have an idea about serving Lua real quick that I will offline email you about.

• Jak

@nrk Any news about that Mongrel configs? :-)

@JakSprats: hehe, too bad I didn't have the time to test thoroughly that beast when it still was without the hypervisor installed (it's a dual Xeon X5660, 2 x 6 cores @ 2.8 GHz). PS: just got your mail, will reply tomorrow.

@agladysh: as soon as I can get that VM up and running again (I don't have access to our VPN right now) and review my hackish installation, I've been busy today but I guess I can get it done over the weekend.

@nrk: OK, thanks, no hurry here :-)