A Simple Webserver Comparison

writeup.md

This is a very simple benchmark comparing the response times of a few different webservers for an extremely simple response: just reply with a snippet of static json. It came up in discussion of a real-life service: in the actual server, a long-running thread/process periodically updates the state from a database, but requests will be served with the data directly from memory. It is imperative, though, that the latencies be extremely low for this service.

This comparison was partly inspired by this blog post.

Method

The code for the various servers may be found here. To conduct each test, I ran the server on a Linux desktop machine and then ran ab (apachebench) against the server from another machine connected via gigabit ethernet on the local network.

Server specs:

• Ubuntu 12.04
• Intel Core i5-2500K CPU (3.30GHz, 4 cores)
• 8GB RAM

For each test, I made 20,000 requests with 1, 10, 100, and 1000 concurrent connections. I did 3 warm-up runs before collecting data. Below are the mean, median, 90th percentile, 99th percentile, and max latencies.

Contenders

I looked at both using Sinatra and writing Rack applications directly. For each of these options, I tested with Racer, Thin, and Unicorn. (This comparison isn't quite fair because I ran Unicorn with 8 workers, whereas Racer and Thin only have 1 worker.) Also in Ruby-land, I tested Goliath.

I was also interested in trying some JRuby servers, so I ran the plain Rack app under Trinidad and mizuno as well.

Additionally, I made a Scala app (using Scalatra) and a Go app (using the net/http standard library package).

Software versions:

• Ruby 1.9.3-p194
• JRuby 1.7.0-rc1
• Scala 2.9.2
• OpenJDK 1.7.0
• Go 1.0.3

Numbers!

The c value is the number of concurrent requests.

• mean: mean query latency (ms)
• median: 50th percentile query latency (ms)
• 90%: 90th percentile query latency (ms)
• 99%: 99th percentile query latency (ms)
• max: maximum query latency (ms)
• qps: mean queries per second.

c = 1

Server	mean	median	90%	99%	max	qps
Sinatra + Racer	0	0	0	1	11	2311
Sinatra + Thin	1	0	1	1	9	1926
Sinatra + Unicorn	1	1	1	7	12	1081
Rack + Racer	0	0	0	0	7	4647
Rack + Thin	0	0	0	0	12	3042
Rack + Unicorn	1	1	1	5	7	1159
Goliath	0	0	0	1	9	2019
JRuby + Mizuno	1	1	1	1	12	1047
JRuby + Trinidad	0	0	0	1	9	2070
Scalatra	1	1	1	1	3	1558
Go	0	0	0	1	4	3754

c = 10

Server	mean	median	90%	99%	max	qps
Sinatra + Racer	2	2	3	8	10	5015
Sinatra + Thin	3	2	3	9	14	3823
Sinatra + Unicorn	1	1	2	17	49	5966
Rack + Racer	1	1	1	2	5	18427
Rack + Thin	1	1	1	4	8	9164
Rack + Unicorn	1	1	2	14	47	7251
Goliath	2	2	2	8	12	4302
JRuby + Mizuno	1	1	2	8	25	6955
JRuby + Trinidad	1	0	1	4	13	16332
Scalatra	3	2	4	5	9	3766
Go	1	1	1	1	3	16730

c = 100

Server	mean	median	90%	99%	max	qps
Sinatra + Racer	18	3	7	12	3618	4735
Sinatra + Thin	25	25	29	30	33	3937
Sinatra + Unicorn	17	15	22	34	47	6008
Rack + Racer	5	1	2	4	1623	12314
Rack + Thin	10	10	13	17	18	9495
Rack + Unicorn	14	13	16	27	42	7357
Goliath	23	24	25	25	33	4398
JRuby + Mizuno	10	5	10	27	1214	8360
JRuby + Trinidad	6	6	7	14	17	16144
Scalatra	22	20	25	30	1016	4557
Go	4	4	5	7	9	23638

c = 1000

Server	mean	median	90%	99%	max	qps
Sinatra + Racer	160	4	1001	2438	6068	3290
Sinatra + Thin	server failed
Sinatra + Unicorn	server failed
Rack + Racer	45	1	3	1460	6459	3095
Rack + Thin	90	13	19	2468	6462	3091
Rack + Unicorn	server failed
Goliath	server failed
JRuby + Mizuno	76	6	29	1229	2462	8010
JRuby + Trinidad	server failed
Scalatra	231	238	273	1150	1414	4153
Go	20	17	21	34	642	20725

Implementation impressions

• At this time, racer seems much more like a proof of concept than a serious production-ready webserver.
• Rack is great, because it's super easy to drop in various webservers to run your app.
• JRuby is really easy to use, and plays well with rbenv and bundler.
• Deployment for JRuby apps may get complex, what with xml files and tomcat configuration and who knows what. Projects like Warbler that turn your whole project into a war file may help a lot though.
• JRuby startup time is really annoying.
• Scala, as usual, is a massive pain to set up and get running. The "minimal" example project for Scalatra required three different tools to set up and configure, and the sbt configuration makes the whole thing a real mess that's very newcomer-unfriendly. There are 13 files in this project, compared with around 2-4 per other implementation.
• I wanted to try Lift as well but the setup was too daunting. sbt is awful.
• Go is really great for this kind of thing. The server is dead-simple, configuration is non-existent, and the app is built to a single binary ready to be deployed to a server.

Conclusions

• Avoid Sinatra if latency on the order of a dozen milliseconds matters more than ease of development.
• Racer has really low latency, but also has a nasty habit of serving a small percentage of requests really slowly as load increases (see c=100, where the rack/racer combo served 99% of requests in 4ms but the max latency was 1623ms).
• Assuming we rule out racer (due to lack of documentation/polish/project momentum), thin seems like the best choice for low-latency performance if you want to stick with MRI.
• Both JRuby servers performed well; Trinidad, in particular, seems like a good bet for low latency. It did stop responding to requests once the concurrent requests got up to 1000, though.
• If you're optimizing for latency, Goliath is a strictly worse choice than, say, Rack + Thin.
• Scalatra is a poor choice for latency-critical applications. This could be due to all the framework code (the same as Sinatra), or because my app was not tuned properly.
• I'd be interested to see how a better Scala app (or a Java app) would perform in the context of a properly-tuned, high-performance Java webserver.
• Go completely dominates this comparison. It would be a great choice for a small standalone webserver like this where latency matters. Note how the implementation doesn't even use any third-party libraries, yet fits into a 30-line file and builds with no extra configuration files whatsoever.