Node JS Benchmarks Amazon AWS vs Joyent

gistfile1.md

The script:

var http = require('http')
var fs = require('fs')


http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'})
  res.end("hello world")
}).listen(3000);

This is just run with node server.js

The local ab

ab -n 100000 -c100 http://localhost:3000/

SmartOS High CPU 1.75 / SW-1

Requests per second: 1339.35 [#/sec] (mean)

Joyent High CPU 1.75 / East-1

Requests per second: 1019.17 [#/sec] (mean)

Joyent High CPU 1.75 Ubuntu 12.04

Requests per second: 2899.45 [#/sec] (mean)

Amazon High CPU 1.75 Ubuntu 12.04

Requests per second: 4085.26 [#/sec] (mean)

Python Loop

The code

for x in xrange(100000000):
    x + 1

The Test

time python test.py

SmartOS High CPU 1.75 / SW-1

real    0m13.531s
user    0m13.487s
sys     0m0.060s

Joyent High CPU 1.75 / East-1

real    0m10.337s
user    0m10.302s
sys     0m0.054s

Joyent High CPU 1.75 Ubuntu 12.04

real    0m9.177s
user    0m9.120s
sys     0m0.040s

Amazon High CPU 1.75

real    0m8.909s
user    0m8.897s                                                                                                          
sys     0m0.012s 

Notes

The first iterations of this had node clustered with 2 forks, incrementing a remote redis counter, and ab being run from a remote machine. While the numbers were higher ( about 50% ) they were proportionally the same. This version just removes all variables.

I understand ab isn't exact but it's margin of error is tight enough to make it suitable for a ballpark. I've worked with Joyent and Amazon in the past. Infact I pulled 100 machines off AWS and put them on Joyent precisely because my own benchmarks - much more thorough than this - showed Joyent being 5 - 10 times faster.

When I ran the above I expected Joyent to completely demolish AWS.

I'm happy to be told I've done something terribly wrong. In fact I'm hoping for it.

Would be interesting with zb instead of ab.

JMeter will allow multiple URL's

zb performs exactly like ab except the reports are prettier. I like the -v mode of zeus enough that if it didn't exhibit the same issue as ab I'd use it.

It demonstrate what Brendan described above. High throughput then kaput. We are moving on to 3 jmeter machines.

One thing to note: Calling ab on Ubuntu or to Ubuntu performs substantially better and with a cleaner spread of connections.

Requests per second: 3618.19 #/sec

I'm guessing this has to do with the TCP implementations.

[admin@d4d38765-6223-6a6f-e9ce-8d5fd6c3abce ~]$ zeusbench  -n 100000 -c 100 -v  http://192.168.25.182:3000/
This is ZeusBench, version 30 (Build date: Jun 20 2013 08:57:11)
   Time   Reqs  Resps |  USucc  UFail | Connecting Connected Out(kb)  In(kb) RespTime Timeouts Bad | Sockets Established Connecting
   1.00   5690   5590 |      0      0 |       5687      5687     355     818    0.017        0   0 |      97          97          0
   2.00   5929   5929 |      0      0 |       5932      5932     370     868    0.017        0   0 |     100         100          0
   3.00   6757   6757 |      0      0 |       6756      6750     421     989    0.014        0   0 |      99          93          6
   4.00   4046   4046 |      0      0 |       4047      3953     247     592    0.008        0   0 |     100           0        100
   5.00   2600   2600 |      0      0 |       2600      2600     162     380    0.002        0   0 |     100           0        100
   6.00   1740   1740 |      0      0 |       1740      1740     108     254    0.001        0   0 |     100           0        100
   7.00   1100   1100 |      0      0 |       1100      1103      68     161    0.001        0   0 |     100           3         97
   8.00    879    879 |      0      0 |        877       877      54     128    0.001        0   0 |      98           1         97
   9.00    856    856 |      0      0 |        857       857      53     125    0.001        0   0 |      99           2         97
  10.00    784    784 |      0      0 |        782       783      48     114    0.001        0   0 |      97           1         96
  11.00    805    805 |      0      0 |        808       804      50     117    0.001        0   0 |     100           0        100
  12.00    885    885 |      0      0 |        885       885      55     129    0.001        0   0 |     100           0        100
  13.00   1113   1113 |      0      0 |       1113      1113      69     163    0.001        0   0 |     100           0        100
  14.00    952    952 |      0      0 |        951       953      59     139    0.001        0   0 |      99           1         98
  15.00    593    593 |      0      0 |        594       593      37      86    0.001        0   0 |     100           1         99
  16.00    678    677 |      0      0 |        678       676      42      99    0.001        0   1 |     100           0        100
  17.00    708    708 |      0      0 |        708       710      44     103    0.001        0   0 |     100           2         98
  18.00    504    504 |      0      0 |        504       502      31      73    0.001        0   0 |     100           0        100
  19.00    749    749 |      0      0 |        749       749      46     109    0.001        0   0 |     100           0        100
  20.00    688    688 |      0      0 |        688       688      43     100    0.001        0   0 |     100           0        100
  21.00    782    782 |      0      0 |        782       782      48     114    0.001        0   0 |     100           0        100
  22.00    586    586 |      0      0 |        585       588      36      85    0.001        0   0 |      99           2         97
  23.00    507    507 |      0      0 |        507       506      31      74    0.001        0   0 |      99           1         98
  24.00    596    596 |      0      0 |        597       595      37      87    0.001        0   0 |     100           0        100
  25.00    635    635 |      0      0 |        635       636      39      93    0.001        0   0 |     100           1         99
  26.00    619    619 |      0      0 |        619       618      38      90    0.001        0   0 |     100           0        100
  27.00    862    862 |      0      0 |        862       862      53     126    0.001        0   0 |     100           0        100
  28.00    935    935 |      0      0 |        935       935      58     136    0.001        0   0 |     100           0        100
  29.00    811    811 |      0      0 |        811       811      50     118    0.001        0   0 |     100           0        100
  30.00    624    624 |      0      0 |        624       627      39      91    0.001        0   0 |     100           3         97
  31.00    545    545 |      0      0 |        543       543      33      79    0.001        0   0 |      98           1         97
  32.00    633    633 |      0      0 |        634       633      39      92    0.001        0   0 |      99           1         98
  33.00    679    679 |      0      0 |        680       679      42      99    0.001        0   0 |     100           1         99
  34.00    764    764 |      0      0 |        764       764      47     111    0.001        0   0 |     100           1         99
  35.00   1001   1001 |      0      0 |       1001      1000      62     146    0.001        0   0 |     100           0        100
  36.00    943    943 |      0      0 |        943       943      58     138    0.001        0   0 |     100           0        100
  37.00    979    979 |      0      0 |        979       980      61     143    0.001        0   0 |     100           1         99
  38.00    786    786 |      0      0 |        785       785      49     115    0.001        0   0 |      99           0         99
  39.00    802    802 |      0      0 |        802       804      50     117    0.001        0   0 |      99           2         97
  40.00    804    804 |      0      0 |        803       803      50     117    0.001        0   0 |      98           1         97
  41.00    862    862 |      0      0 |        863       861      53     126    0.001        0   0 |      99           0         99
  42.00    857    857 |      0      0 |        858       857      53     125    0.001        0   0 |     100           0        100
  43.00    994    993 |      0      0 |        994       993      62     145    0.001        0   1 |     100           0        100
  44.00   1103   1103 |      0      0 |       1103      1103      68     161    0.001        0   0 |     100           0        100
  45.00    946    942 |      0      0 |        944       947      59     138    0.001        0   4 |      98           5         93
  46.00    549    548 |      0      0 |        551       544      33      80    0.001        0   1 |     100           1         99
  47.00    670    669 |      0      0 |        669       669      41      98    0.001        0   1 |      99           1         98
  48.00    689    689 |      0      0 |        690       688      42     100    0.001        0   0 |     100           0        100
  49.00    706    704 |      0      0 |        706       704      43     103    0.001        0   2 |     100           0        100
  50.00    706    706 |      0      0 |        706       706      44     103    0.001        0   0 |     100           0        100
  51.00    794    789 |      0      0 |        794       789      49     115    0.001        0   5 |     100           0        100
  52.00    814    811 |      0      0 |        814       811      50     118    0.001        0   3 |     100           0        100
  53.00    697    697 |      0      0 |        695       699      43     102    0.001        0   0 |      98           2         96
  54.00    674    672 |      0      0 |        673       671      41      98    0.001        0   2 |      97           1         96
  55.00    664    662 |      0      0 |        665       661      41      96    0.001        0   2 |      98           0         98
  56.00    663    662 |      0      0 |        665       664      41      96    0.001        0   1 |     100           2         98
  57.00    465    465 |      0      0 |        465       463      28      68    0.001        0   0 |     100           0        100
  58.00    583    580 |      0      0 |        583       580      36      84    0.001        0   3 |     100           0        100
  59.00    682    679 |      0      0 |        682       679      42      99    0.001        0   3 |     100           0        100
  60.00    580    580 |      0      0 |        579       580      36      84    0.001        0   0 |      99           0         99
  61.00    458    457 |      0      0 |        459       458      28      66    0.001        0   1 |     100           1         99
   Time   Reqs  Resps |  USucc  UFail | Connecting Connected Out(kb)  In(kb) RespTime Timeouts Bad | Sockets Established Connecting
  62.00    372    372 |      0      0 |        372       371      23      54    0.001        0   0 |     100           0        100
  63.00    338    337 |      0      0 |        337       338      21      49    0.001        0   1 |      99           1         98
  64.00    243    243 |      0      0 |        244       242      15      35    0.001        0   0 |     100           0        100
  65.00    308    308 |      0      0 |        308       308      19      45    0.001        0   0 |     100           0        100
  66.00   1264   1264 |      0      0 |       1264      1264      79     185    0.001        0   0 |     100           0        100
  67.00   1861   1861 |      0      0 |       1861      1861     116     272    0.001        0   0 |     100           0        100
  68.00   1978   1978 |      0      0 |       1977      1979     123     289    0.001        0   0 |      99           1         98
  69.00   1586   1585 |      0      0 |       1587      1591      99     232    0.001        0   1 |     100           7         93
  70.00   1664   1663 |      0      0 |       1663      1657     103     243    0.001        0   1 |      99           1         98
  71.00   1928   1928 |      0      0 |       1929      1927     120     282    0.001        0   0 |     100           0        100
  72.00   2147   2147 |      0      0 |       2146      2148     134     314    0.001        0   0 |      99           1         98
  73.00   1686   1685 |      0      0 |       1687      1684     105     246    0.001        0   1 |     100           0        100
  74.00   1138   1138 |      0      0 |       1138      1138      71     166    0.001        0   0 |     100           0        100
  75.00   1146   1146 |      0      0 |       1143      1148      71     167    0.001        0   0 |      97           2         95
  76.00   1003   1003 |      0      0 |       1004      1001      62     146    0.001        0   0 |      98           0         98
  77.00   1514   1514 |      0      0 |       1516      1515      94     221    0.001        0   1 |      99           1         98
  78.00   1108   1108 |      0      0 |       1105      1109      69     162    0.001        0   0 |      96           2         94
  79.00    842    844 |      0      0 |        845       842      52     123    0.001        0   0 |      97           0         97
  80.00    819    818 |      0      0 |        819       818      51     119    0.001        0   1 |      97           0         97
  81.00   1277   1276 |      0      0 |       1277      1276      79     186    0.001        0   1 |      97           0         97
  82.00   1076   1076 |      0      0 |       1076      1077      67     157    0.001        0   0 |      97           1         96
  83.00    813    814 |      0      0 |        812       813      50     119    0.001        0   0 |      95           0         95
  84.00    487    488 |      0      0 |        488       488      30      71    0.001        0   0 |      95           0         95
  85.00    416    416 |      0      0 |        416       418      26      60    0.001        0   0 |      95           2         93
  86.00    705    705 |      0      0 |        705       704      44     103    0.001        0   0 |      95           1         94
  87.00    760    761 |      0      0 |        759       761      47     111    0.001        0   0 |      93           1         92
  88.00    632    632 |      0      0 |        633       631      39      92    0.001        0   0 |      94           0         94
  89.00    551    551 |      0      0 |        551       551      34      80    0.001        0   0 |      94           0         94
  90.00    376    376 |      0      0 |        376       376      23      55    0.001        0   0 |      94           0         94
  91.00    531    533 |      0      0 |        531       533      33      78    0.001        0   0 |      92           0         92
  92.00    348    350 |      0      0 |        347       350      21      51    0.001        0   0 |      89           0         89
  93.00    287    287 |      0      0 |        288       288      17      41    0.001        0   0 |      90           1         89
  94.00    289    291 |      0      0 |        289       290      18      42    0.001        0   0 |      88           0         88
  95.00    116    117 |      0      0 |        116       118       7      17    0.001        0   0 |      87           1         86
  96.00    242    244 |      0      0 |        242       244      15      35    0.001        0   0 |      85           1         84


Requests made:        100084
HTTP responses:       99963
Sent:                 6397760 [Bytes]
Received:             14994450 [Bytes]
Keepalives:           0
Keepalives dropped:   0
SSL sessions reused:  0
Connections made:     99965
Connections dropped:  0
Invalid responses:    0
Failed connections:   37
Timeouts (connect):   0
Timeouts (data):      0
Size mis-matches:     0
Early responses:      0
Max concurrency:      0

Run time:             88.763 [seconds] (0.2 user, 0.9 system)
Request  rate:        1127.542 [requests/sec]
Response rate:        1126.179 [responses/sec]
Transfer rate:        164.968 [KBytes/sec] received
Transfer rate (out):  70.388 [KBytes/sec] sent
Average response:     0.004 [seconds]
Average transaction:  0.004 [seconds]

Response codes:
               200:   99963

*** Test didn't seem to run successfully ***
There were failed connections

We cut the benchmarks over to jmeter. 3 servers at a concurrency of 33.

var http = require('http')
var redis = require("redis")


var client = redis.createClient(6379,"x.x.x.x" )
var body = "hello world"

    http.createServer(function (req, res) {
      res.writeHead(200, {'Content-Type': 'text/plain', 'Content-Length': body.length})
      client.incr("foo", function (err, data) { res.end("hello world") } );
    }).listen(3000);

Joyent SmartOS 1.75 GB High CPU

Requests per second: 7029

Joyent High CPU 1.75 Ubuntu 12.04

Requests per second: 4800

Amazon numbers are the same.

Thank god I don't have to use AWS!

I'd be interested to see how wrk performs here too.

Also, just for anyone else who's tempted to copy/paste the Content-Length snippet - it should technically be 'Content-Length': Buffer.byteLength(body) - have been bitten by this before!

@mhart. Nice catch. Copy Paste is exactly what I did.

wrk looks interesting. Though it looks like it runs under the same paradigm as ab and zeusbench. One process on one machine opening n connections.

What we found was that the issue wasn't the benchmarking tool. ab and zb are booth good tools for what they do. The issue is in the the SmartOS TCP stack, it doesn't like getting a ton of requests from a single ip address. In order to get a reliable benchmark we had to reduce the concurrency and distribute the benchmark clients across multiple machines. 3 machines running at a concurrency of 33. jmeter, funkload, and tsung should all be able to perform this reasonably well. One could in theory fire ab, zb, wrk on muliple machines simultaneously and aggregate the results but the other tools I list already have that built in.