Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Phoenix Showdown Comparative Benchmarks @ Rackspace

Comparative Benchmark Numbers @ Rackspace

I've taken the benchmarks from Matthew Rothenberg's phoenix-showdown, updated Phoenix to 0.13.1 and ran the tests on the most powerful machines available at Rackspace.

Results

Framework Throughput (req/s) Latency (ms) Consistency (σ ms)
Plug 198328.21 0.63ms 2.22ms
Phoenix 0.13.1 179685.94 0.61ms 1.04ms
Gin 176156.41 0.65ms 0.57ms
Play 171236.03 1.89ms 14.17ms
Phoenix 0.9.0-dev 169030.24 0.59ms 0.30ms
Express Cluster 92064.94 1.24ms 1.07ms
Martini 32077.24 3.35ms 2.52ms
Sinatra 30561.95 3.50ms 2.53ms
Rails 11903.48 8.50ms 4.07ms

Environment

Environment Version
Erlang 17.5
Elixir 1.1.0-dev
Ruby 2.2.0
Node 0.12.4
Go 1.3.3
Java 8u45

Server

Rackspace OnMetal IO
CPU Dual 2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	128 GB
System Disk	32 GB
Data Disk	Dual 1.6 TB PCIe flash cards
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$2.4658 / hr

Client

Rackspace OnMetal Compute
CPU	2.8 Ghz, 10 core Intel® Xeon® E5-2680 v2
RAM	32 GB
System Disk	32 GB
Network	Redundant 10 Gb / s connections in a high availability bond
Disk I/O	Good
Price	$0.7534 / hr

Detailed Results

PHOENIX 0.9.0-dev (original)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   595.50us  398.08us  62.05ms   99.05%
    Req/Sec     8.50k   304.25     9.20k    70.34%
  5087667 requests in 30.10s, 10.42GB read
Requests/sec: 169030.24
Transfer/sec:    354.47MB

PHOENIX 0.13.1 (current)

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.13.193:4000/showdown"
Running 30s test @ http://10.184.13.193:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   611.61us    1.04ms  95.57ms   99.05%
    Req/Sec     9.04k   656.30    10.98k    71.84%
  5408349 requests in 30.10s, 11.08GB read
Requests/sec: 179685.94
Transfer/sec:    376.82MB

PLUG

server:~# MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix server
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:4000/showdown"
Running 30s test @ http://10.184.11.239:4000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   631.83us    2.22ms 217.98ms   99.05%
    Req/Sec     9.99k   761.02    18.66k    78.99%
  5969613 requests in 30.10s, 12.00GB read
Requests/sec: 198328.21
Transfer/sec:    408.34MB

PLAY

server:~# ./activator clean stage
server:~# ./target/universal/stage/bin/play-scala -Dhttp.port=5000
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:5000/showdown"
Running 30s test @ http://10.184.11.239:5000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.89ms   14.17ms 349.18ms   98.57%
    Req/Sec     8.80k     1.58k   23.45k    86.04%
  5154091 requests in 30.10s, 9.82GB read
Requests/sec: 171236.03
Transfer/sec:    334.12MB

RAILS

server:~# PUMA_WORKERS=20 MIN_THREADS=1 MAX_THREADS=80 RACK_ENV=production bundle exec puma
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.50ms    4.07ms  61.71ms   70.94%
    Req/Sec   598.10     40.81   740.00     71.03%
  357320 requests in 30.02s, 788.91MB read
Requests/sec:  11903.48
Transfer/sec:     26.28MB

SINATRA

server:~# RACK_ENV=production bundle exec puma -t 1:16 -w 20 --preload
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:9292/showdown"
Running 30s test @ http://10.184.11.239:9292/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.50ms    2.53ms  56.23ms   74.62%
    Req/Sec     1.54k   414.73     5.33k    65.96%
  919897 requests in 30.10s, 1.85GB read
Requests/sec:  30561.95
Transfer/sec:     63.04MB

GIN

server:~# GOMAXPROCS=20 GIN_MODE=release go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   655.59us  573.14us  10.11ms   89.21%
    Req/Sec     8.86k   759.03    15.02k    82.84%
  5302230 requests in 30.10s, 10.31GB read
Requests/sec: 176156.41
Transfer/sec:    350.61MB

MARTINI

server:~# GOMAXPROCS=20 MARTINI_ENV=production go run server.go
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.35ms    2.52ms  32.73ms   60.45%
    Req/Sec     1.61k    99.91     1.98k    68.77%
  962648 requests in 30.01s, 1.87GB read
Requests/sec:  32077.24
Transfer/sec:     63.84MB

EXPRESS (20-node cluster)

server:~# NODE_ENV=production node server.js -w 20
client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.184.11.239:3000/showdown"
Running 30s test @ http://10.184.11.239:3000/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.24ms    1.07ms  26.92ms   83.39%
    Req/Sec     4.63k     2.37k   13.34k    74.60%
  2771185 requests in 30.10s, 5.65GB read
Requests/sec:  92064.94
Transfer/sec:    192.37MB

Bonus Track

ASP.Net 4.5.1 MVC 5 on IIS 8

I've benchmarked ASP.Net, since I do .Net at my day to day job. Rackspace doesn't offer an OnMetal instance with windows, only virtualized instances, so I've done the benchmarks on the most powerful one I could find.

On a Rackspace I/O v1 120GB instance is listed as:

CPU:	32 vCPUs
RAM:	120 GB
System Disk:	40 GB
Data Disk:	1.2 TB (4 disks)
Network:	10 Gb / s
Disk I/O:	Best

Results:

root@client:~# wrk -t20 -c100 -d30S --timeout 2000 "http://10.223.241.211/showdown"
Running 30s test @ http://10.223.241.211/showdown
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    62.98ms  104.13ms 962.77ms   81.84%
    Req/Sec     1.02k   435.75     2.92k    70.76%
  581748 requests in 30.02s, 1.39GB read
Requests/sec:  19381.05
Transfer/sec:     47.37MB

Results were pretty horrid, but maybe it's the virtualization overhead, or maybe Rackspace's Windows virtualization is bad, or even the 10gbit network between OnMetal instances and virtualized instances is not that good. I might do more testing this weekend to try to figure that out.

zepplock commented Jul 9, 2015

Would be great to see updated .NET results ;)

Owner

omnibs commented Jul 20, 2015

I've been busy lately, but I plan on running Windows + Linux benches on Amazon. Amazon has that whole deal with network drivers for 10gbit to work so I've been putting it off heh

I did run linux benchmarks on the same machine that got me 19k on ASP.Net and results were not amazing at all. Top performers did around 80k req/s. I guess virtualization really sucks (yay for OnMetal).

bbhoss commented Sep 15, 2015

I think benchmarking things on AWS is pretty pointless. You never know what hardware you're going to get or how utilized it is. No test run has a chance at ever being even similar to others, unless you're running on dedicated hardware.

Please update Phoenix to the latest stable, since this benchmark is using beta version of Phoenix

What does exactly the Consistency column mean?

peteygao commented Apr 3, 2016

Consistency refers to Standard Deviation (as per the Greek letter Sigma: σ). Roughly speaking, 68% of all requests fall within the stated "deviation" in terms of milliseconds. More about Standard Deviation on Wikipedia.

naterh commented Jul 3, 2016

I would be curious to see the same tests ran on Rackspace's new-ish OnMetal v2 servers that come with better virt networking.

Owner

omnibs commented Jul 13, 2016

Didn't know they had better network on v2, interesting. @naterh do you have a source on that? All I found was about some VPN-ish hybrid cloud stuff.

Honestly I just saw the specs had lower clocks and didn't bother playing with them haha

Can you use a monospace font in the table for numbers? Thanks!

kiopro commented Sep 26, 2016

try iris for golang https://github.com/kataras/iris :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment