Benchmarking Hops with various configurations (spoiler: Mongrel2 + LuaJIT 2 == FAST!)

hops_lighttpd_fcgi_lua51_maxprocs1.txt

$ ab -n 1000 -c 50 http://10.1.1.82:80/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        lighttpd/1.4.28
Server Hostname:        10.1.1.82
Server Port:            80

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.488 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      418418 bytes
HTML transferred:       257257 bytes
Requests per second:    2048.74 [#/sec] (mean)
Time per request:       24.405 [ms] (mean)
Time per request:       0.488 [ms] (mean, across all concurrent requests)
Transfer rate:          837.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       4
Processing:     2   23   2.7     23      27
Waiting:        2   23   2.7     23      27
Total:          7   24   2.2     24      30
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     24
  66%     24
  75%     25
  80%     25
  90%     25
  95%     26
  98%     26
  99%     26
 100%     30 (longest request)

hops_lighttpd_fcgi_luajit2_maxprocs1.txt

$ ab -n 1000 -c 50 http://10.1.1.82:80/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        lighttpd/1.4.28
Server Hostname:        10.1.1.82
Server Port:            80

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.371 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      418418 bytes
HTML transferred:       257257 bytes
Requests per second:    2691.84 [#/sec] (mean)
Time per request:       18.575 [ms] (mean)
Time per request:       0.371 [ms] (mean, across all concurrent requests)
Transfer rate:          1099.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.9      1       5
Processing:     4   17   1.9     18      20
Waiting:        4   17   1.9     18      20
Total:         10   18   1.3     18      23

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     18
  75%     19
  80%     19
  90%     19
  95%     20
  98%     20
  99%     22
 100%     23 (longest request)

hops_xavante_lua51.txt

$ ab -n 1000 -c 50 http://10.1.1.82:6789/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Xavante
Server Hostname:        10.1.1.82
Server Port:            6789

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.699 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      397000 bytes
HTML transferred:       257000 bytes
Requests per second:    1431.30 [#/sec] (mean)
Time per request:       34.933 [ms] (mean)
Time per request:       0.699 [ms] (mean, across all concurrent requests)
Transfer rate:          554.91 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       5
Processing:     3   21  17.2     18     224
Waiting:        3   20  17.3     18     224
Total:          7   22  17.2     19     225
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     20
  75%     20
  80%     21
  90%     22
  95%     29
  98%     63
  99%     63
 100%    225 (longest request)

hops_xavante_luajit2.txt

$ ab -n 1000 -c 50 http://10.1.1.82:6789/hello/world
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.82 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Xavante
Server Hostname:        10.1.1.82
Server Port:            6789

Document Path:          /hello/world
Document Length:        257 bytes

Concurrency Level:      50
Time taken for tests:   0.383 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      397000 bytes
HTML transferred:       257000 bytes
Requests per second:    2613.76 [#/sec] (mean)
Time per request:       19.130 [ms] (mean)
Time per request:       0.383 [ms] (mean, across all concurrent requests)
Transfer rate:          1013.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      0       4
Processing:     2   13  11.3     12     216
Waiting:        2   12  11.3     12     216
Total:          6   14  11.5     13     219
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     13
  66%     13
  75%     14
  80%     14
  90%     14
  95%     17
  98%     23
  99%     24
 100%    219 (longest request)

lighttpd.conf

# This is the most basic lighttpd.conf (derived from the default one shipped with
# the Debian package of Lighttpd) that can be used to test Hops with FastCGI.

server.modules = (
    "mod_access",
    "mod_fastcgi",
    "mod_rewrite",
)

# Replace with the full path to the 'example' directory of Hops.
server.document-root  =  "/home/adaniele/development/hops/example"
server.errorlog       = "/var/log/lighttpd/error.log"
server.pid-file       = "/var/run/lighttpd.pid"
server.username       = "www-data"
server.groupname      = "www-data"
url.access-deny       = ( "~", ".inc" )
static-file.exclude-extensions = ( ".fcgi" )

fastcgi.server = (
    ".lua" => (
        "localhost" => (
            "socket"   => "/tmp/lua-fastcgi_hops.socket",

            # Replace with the actual path on your system where the
            # wsapi.fcgi file (provided by wsapi-fcgi) is located.
            # NOTE: to switch between Lua 5.1 and Luajit2 you can edit
            # wsapi.fcgi so that it points to a different executable
            # used by the 'exec' command.
            "bin-path" => "/home/adaniele/.luarocks/bin/wsapi.fcgi",

            # Specify how many Lua processes are used by FastCGI
            # to elaborate requests. Setting max-procs to 1 means
            # using just one Lua process.
            "min-procs" => 1,
            "max-procs" => 1,
        )
    )
)

url.rewrite-once = (
    "^/([^.]+)?$" => "/app.lua/$1",
)

Some benchmarks for Hops using various configurations. Hops is executed on a virtualized (VMware ESX) Debian Squeeze with only two CPUs assigned and ab is launched on a completely different host machine connected with a gigabit link (same network). A couple of additional notes:

Xavante: the reload parameter is set to false to prevent the app from reloading on each request.
Lighttpd: the fastcgi.server backend for Lua is configured with the max-procs directive set to 1 (only one Lua process is used) so that we can obtain results that can be compared directly with the benchmarks that use Xavante. Just for reference, Lighttpd with a plain HTML file can reach an average of 8026.71 req/sec.

As one would expect, Lighttpd can yield much better results just by setting max-procs to 2:

Lua 5.1: 3590.84 #/sec
Luajit2: 3490.41 #/sec

Strangely enough Luajit2 isn't faster than Lua 5.1 here. I guess it's just because the VM is configured with only 2 CPUs, even if the actual load of the server does not even reach 80% during these tests.

This is awesome! Any chance you could attach your lighttpd config for anyone else that might want to try?

Sure, I reduced the whole lighttpd.conf so that only the basic stuff needed to get Hops up and running is there, but the relevant bits can be easily extracted to have it configured on an existing lighttpd install using virtual hosts.

~2K RPS per thread is to be expected for wsapi+luajit2 on noop handlers.

Good stuff. At least it looks like Hops isn't doing anything insanely stupid to ruin performance. :)

A few more tests, this time using Mongrel2 (via mongrel2_wsapi) and producing numbers that should be compared with Lighttpd configured with max-procs > 1 max-procs = 1. Please note that I had to use http_load instead of ab since the latter does not play nice with Mongrel2.

The difference between the standard Lua interpreter and Luajit2 is much more enjoyable here, going from great to awesome numbers.

Using Lua 5.1 - http://10.1.1.82:8080/hello/world

$ ./http_load -verbose -parallel 50 -fetches 1000 mongrel2.url
1000 fetches, 50 max parallel, 257000 bytes, in 0.342902 seconds
257 mean bytes/connection
2916.29 fetches/sec, 749485 bytes/sec
msecs/connect: 0.435044 mean, 1.464 max, 0.218 min
msecs/first-response: 16.1858 mean, 17.671 max, 2.188 min
HTTP response codes:
  code 200 -- 1000

Using luajit2 - http://10.1.1.82:8080/hello/world

$ ./http_load -verbose -parallel 50 -fetches 1000 mongrel2.url
1000 fetches, 50 max parallel, 257000 bytes, in 0.222126 seconds
257 mean bytes/connection
4501.95 fetches/sec, 1.157e+06 bytes/sec
msecs/connect: 0.49287 mean, 2.115 max, 0.21 min
msecs/first-response: 10.1419 mean, 12.154 max, 1.789 min
HTTP response codes:
  code 200 -- 1000

Good stuff!

Hey @norman I updated my comment about the performance measurements using mongrel2 since I was inadvertently running my tests with only 1 backend process... so yes, 4501 req/sec with only one luajit2 process. With 2 luajit2 backend processes, the benchmark reaches 7689.87 req/sec. Awesome stuff.

So, is it Mongrel2 so much faster than lighty? It maybe worth for me to switch my engine from nginx/spawn-fcgi to it...

@nrk: can you point me somewhere where it is written how to setup mongrel with wsapi?

@agladysh I reconfigured my virtual machine to use 4 CPUs and set max-procs = 3 in the lighttpd.conf: with this configuration I can get up to 6052.38 fetches/sec using Luajit2, which is a nice improvement but it's still far from the results that I can get with 2 processes of Luajit2 and Mongrel2. By the way, Mongrel2 with 3 Luajit2 processes easily tops 8100 req/sec, but I guess it's reaching its limit for that configuration.

So, unless I got something horribly wrong, I'd say that Mongrel2 is indeed damn fast. I still haven't tried with nginx, but it's up next.

As for how to get it up and running, I didn't follow any instruction but it's actually quite easy (at least without proper process management), you just need mongrel2-lua and mongrel2_wsapi and a few lines of Lua for additional configuration. I will post a step-by-step by tomorrow, just in case.

@nrk: nginx should be about the same speed as lighty, but it worth benchmarking.

step-by-step guide would be most useful (maybe you can post it somewhere on the web as a standalone document, not as comment?) :-)

@agladysh yup, will use a separate gist for now so that users can also commit their corrections :-)

Thanks for the benchmarks and info - great to see you guys playing around with my code. :) BTW I'm pretty sure Orbit, Mercury and other web frameworks would have similar performance though too, there's nothing too special about Hops in terms of what it does to be fast. Really the credit goes entirely to Lua/LuaJIT and WSAPI.

Well yeah, it actually turned into a quick and nice demonstration about how the Mongrel2+Luajit2 pair can be really fast allowing for an high number of RPS with a very low memory footprint and CPU usage. Sticking with Hops as a reference microframework for these tests is now a matter of consistency, maybe I'll do the same with other frameworks such as Orbit just out of curiosity :-) Anyway, nice work on Hops!

Hi Danielle,
this all kicks ass ... where is it heading to, are you gonna make something other people can easily re-use .. I ask because I am becoming more and more of a Lua-junky.

Some questions:
1.) why not run this on bare metal?
2.) use 4 cores, or 6 cores - get some BIG benchmark numbers (like 30-50K)
3.) please please plug this all into a non-blocking redis backend framework ... please please :)

I hope all of this stuff results in some sort of real framework, something I can just be lazy and use, and all this work, you should write it up in a blog, cause Lua needs the press, its the absolute shit and people just dont know about it.

nice work ... keep going

• jak

Thanks! Glad you're liking Hops. I'm currently working on better template handling (caching, responding with different templates based on content type headers, the request and extension like Rails), but other than that the focus is more on ease of use than high performance. A certain good level of performance is guaranteed just by using Lua and WSAPI.

If you're looking for a non-blocking web development API for Lua then you might want to check out LuaNode.

@JakSprats: hey Jak it's been a while, and it's good to know that you are enjoying Lua more and more. In reply to your questions:

1. the reason is simple: we, at work, rely heavily on our in-house virtualization infrastructure. We still have a few bare-metal installations, but that hardware is relatively old.
2. I can still configure my virtual machine to use 8 cores :-) That ESX host is brand new and still unused aside from some test stuff, so there's actually nothing running while I do the tests.
3. I don't know how and if a WSAPI-based application can behave with non-blocking stuff, but aside from that I'd love to make redis-lua completely async with the next major version. As @norman pointed out you should definitely take a look at LuaNode.

@norman: Jak and performances go hand in hand ;-)

@nrk hey Danielle, 1 & 2 were just me wanting to see higher numbers, I can do the math in my head already, and this platform would smoke on a 6 core @3.2Ghz :)
I asked Ignacio of LuaNode about 3 months ago about redis integration and he said: no time right now.
So the non-blocking Lua-webserver w/ nonblocking redis backend remains to-be-done (redis2-nginx-module has done it, but I remember you wanted it done differently than they did it).
Anyways, I am polluting this gist, I have an idea about serving Lua real quick that I will offline email you about.

• Jak

@nrk Any news about that Mongrel configs? :-)

@JakSprats: hehe, too bad I didn't have the time to test thoroughly that beast when it still was without the hypervisor installed (it's a dual Xeon X5660, 2 x 6 cores @ 2.8 GHz). PS: just got your mail, will reply tomorrow.

@agladysh: as soon as I can get that VM up and running again (I don't have access to our VPN right now) and review my hackish installation, I've been busy today but I guess I can get it done over the weekend.

@nrk: OK, thanks, no hurry here :-)