Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Hello world of Sinatra-like libraries on Elixir, Ruby and Node.js

Lies, Damned Lies and Benchmarks

This is just an exercise to measure the performance between Sinatra-like libraries in:

The basic example uses only a single route at /hello_world and return Hello world. These benchmarks are extremely simple and should not be used as a decision factor for any of the tools presented (unless you are building hello world apps ;). I am just using them as performance guideline when working on Dynamo.

What happens if we add more routes?

This benchmark also measures what happens if you add 10, 100 and 1000 /hello_world/#{i} routes and try to match the last /hello_world/1000 route.

What happens if I am serving a file?

We also check the behavior of the app when it has to serve a file with 100kb of size.

What happens if I implement a Fibonacci server?

Just trolling bro.

Results

Results were obtained by running the examples below in a Macbook Pro 2011 2.3GHz dual-core Intel Core i5. No customization was done to the underlying VM/Engine/Webserver besides passing the "production" flag to all libraries, i.e. I am measuring the out-of-the-box experience. This means Elixir is running on both cores, while the other solutions aren't. This is one of the several benefits of using the Erlang VM.

req/s average for 8000 requests, concurrency 40 (ab -n 8000 -c 40) (more is better):

   Routes     | Dynamo/Elixir | SinatraS/Ruby 1.9.2 | Express/node.js 0.4.7
      1       |    9135.83    |      3833.54        |      4730.74
   10 + 1     |    9105.35    |      3561.69        |      4635.8
  100 + 1     |    9103.91    |      2472.51        |      4171.94
 1000 + 1     |    9043.72    |       598.47        |      1904.71
 100kb file   |    6087.99    |      2088.66        |      1734.17

Notes

Thanks to the Erlang VM, Dynamo performs very well in the basic hello world case. Indeed, Dynamo is in very early stage and the throughput will reduce by a couple dozens req/s as we add more basic features. Regardless, very good results.

It is also important to notice that increasing the number of routes increases the CPU usage which will block on single process, non threaded solutions like Node.js, affecting throughput. Also the performance of the algorithm used in the routes matching matters in the results above.

In the future, I plan to draw graphs on how those examples behave when increasing the concurrency. In early experiments, both Sinatra and Express throughput starts to decrease early on when increasing the concurrency. I also want to run benchmarks against those servers deployed on Heroku, but not sure if i need their permission first or anything. :)

defmodule MyApp do
use Dynamo::Routes
get "/hello_world" do
"Hello World"
end
end
Dynamo::Cowboy.run MyApp
require 'sinatra/base'
require 'sinatra/synchrony'
class HelloWorld < Sinatra::Base
register Sinatra::Synchrony
get '/hello_world' do
'Hello World'
end
end
var app = require('express').createServer();
app.get('/hello_world', function(req, res){
res.send('hello world');
});
app.listen(3000);
@sebastiansito

This comment has been minimized.

Copy link

@sebastiansito sebastiansito commented Mar 2, 2012

Promising results. Are You planning to prepare whole non-blocking stdlib for Elixir?
What is Dynamo? The README isn't descriptive.
Numbers are OK for regular apps but if You can lift them even more it could be agile competitor for libevent or for example LMAX Disruptor on APIs and services field.

@josevalim

This comment has been minimized.

Copy link
Owner Author

@josevalim josevalim commented Mar 2, 2012

@rafaelrosafu

This comment has been minimized.

Copy link

@rafaelrosafu rafaelrosafu commented Mar 2, 2012

Is there a typo on the benchmarks for Sinatra with a 1000 + 1? It's displaying 598.47, is that correct? Besides that, perhaps you should put a "higher numbers are better" message for the casual readers :)

@josevalim

This comment has been minimized.

Copy link
Owner Author

@josevalim josevalim commented Mar 2, 2012

Not a typo. :) And I added a "more is better" note. Thanks!

@sebastiansito

This comment has been minimized.

Copy link

@sebastiansito sebastiansito commented Mar 8, 2012

Also, in many cases ab is broken on OS X. I found worth trying to run benchmarks on linux (I'm using OSX Lion by default) which leads to different results. Probably You have good version of ab but to be real sure about your numbers, what was complete / failed requests ratio?

@josevalim

This comment has been minimized.

Copy link
Owner Author

@josevalim josevalim commented Mar 8, 2012

@burnhype I don't have this information anymore. I hope to re-run those benches in the future.

@fredwu

This comment has been minimized.

Copy link

@fredwu fredwu commented Jun 30, 2013

@josevalim Hey Jose, any chance of a new benchmark for the latest versions? :)

@abhijitiitr

This comment has been minimized.

Copy link

@abhijitiitr abhijitiitr commented May 4, 2014

@josevalim I think NodeJS would perform better than Elixir after clustering.

@wess

This comment has been minimized.

Copy link

@wess wess commented May 23, 2014

@abhijitiitr - Defeats the purpose of out of box if he has to go add a bunch of extensions, no?

@abhijitiitr

This comment has been minimized.

Copy link

@abhijitiitr abhijitiitr commented May 31, 2014

@wess it does...but does it really matter since clustering is commonly used in all nodejs apps and has become part of the standard code in nodejs

@erickhsiao

This comment has been minimized.

Copy link

@erickhsiao erickhsiao commented Jul 1, 2014

@abhijitiitr if 1 instance of elixir can beat 1 instance of node.js in any scenario, why would cluster node.js perform better than elixir?

@jocull

This comment has been minimized.

Copy link

@jocull jocull commented Jan 15, 2018

Um, I actually do have a Node+Express Fibonacci service... :trollface:
https://github.com/jocull/node-fibonacci-service

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.