RexProTimings.md

Results from Fabric Timing tests. I'm trying to establish a performance baseline, and generally, to better understand the available methods for executing Gremlin scripts from outside the JVM.

The tests each execute a simple g Gremlin script (to obtain the graph object) using different methods. My testing scenario uses Titan (+ local Berekely DB) on a separate machine in the same local network. The RexProClient library for these tests was built in Release mode.

Update (2013-06-07): I have improved the tests based on suggestions from Daniel. See updated results below.

Rest API (Gremlin extension)

Wiki Page

RunRestApi:    0.0115ms,    5.5213ms,    5.5248ms,    5.5433ms
RunRestApi:    0.0080ms,    5.0159ms,    5.0208ms,    5.1927ms
RunRestApi:    0.0069ms,    4.6975ms,    4.7017ms,    4.7160ms
RunRestApi:    0.0069ms,    4.2712ms,    4.2744ms,    4.2898ms
RunRestApi:    0.0073ms,    5.6557ms,    5.6599ms,    5.6774ms
RunRestApi:    0.0083ms,    9.1399ms,    9.1437ms,    9.1602ms
RunRestApi:    0.0073ms,    5.1332ms,    5.1377ms,    5.2298ms
RunRestApi:    0.0104ms,    4.2492ms,    4.2530ms,    4.3685ms
RunRestApi:    0.0090ms,    4.7783ms,    4.7836ms,    4.8028ms
RunRestApi:    0.0083ms,    4.4354ms,    4.4399ms,    4.5617ms
AVG: 5.35422ms

RexConnect

Project Site

RunRexConn:    0.0020ms,    0.0139ms,    1.3270ms,   10.5418ms
RunRexConn:    0.0024ms,    0.0157ms,    1.7372ms,    9.8598ms
RunRexConn:    0.0024ms,    0.0125ms,    2.2222ms,   11.2631ms
RunRexConn:    0.0017ms,    0.0136ms,    1.4803ms,    9.5826ms
RunRexConn:    0.0017ms,    0.0101ms,    1.1583ms,   10.1814ms
RunRexConn:    0.0017ms,    0.0104ms,    1.2227ms,    8.4431ms
RunRexConn:    0.0017ms,    0.0097ms,    1.2122ms,   20.7232ms
RunRexConn:    0.0027ms,    0.0185ms,    1.5076ms,    8.5939ms
RunRexConn:    0.0017ms,    0.0094ms,    1.2878ms,    8.6307ms
RunRexConn:    0.0020ms,    0.0139ms,    1.3204ms,   10.2930ms
AVG: 10.81126ms

The "query time" that RexConnect measures is very close to the Gremlin extension timings (usually between 5-7ms). When I measured the JSON parsing + writing to the FilterChain, it was always 0-1ms. Removing the logging output didn't make any noticeable performance difference.

This implies that RexConnect's extra ~5ms is not RexPro's fault. It could be the JSON command processing that RexConnect does, or perhaps the way RexConnect uses Grizzly filters.

RexPro Client

Project Site

RunRexPro:    0.0003ms,    0.0024ms,   27.8894ms,   27.9227ms
RunRexPro:    0.0045ms,    0.0062ms,   31.5273ms,   31.5696ms
RunRexPro:    0.0003ms,    0.0024ms,   31.2851ms,   31.3159ms
RunRexPro:    0.0003ms,    0.0027ms,   28.6968ms,   28.7297ms
RunRexPro:    0.0003ms,    0.0024ms,   29.7372ms,   29.7711ms
RunRexPro:    0.0003ms,    0.0027ms,   32.6993ms,   32.7322ms
RunRexPro:    0.0003ms,    0.0027ms,   28.1365ms,   28.1711ms
RunRexPro:    0.0000ms,    0.0020ms,   27.8565ms,   27.8891ms
RunRexPro:    0.0003ms,    0.0020ms,   28.8889ms,   28.9204ms
RunRexPro:    0.0003ms,    0.0024ms,   29.4611ms,   29.4964ms
AVG: 29.65182ms

RexProTimings2.md

I added some parallel test cases (in this commit). Latest results:

Rest API (Gremlin extension)

Serial

RunRestApi:    0.0083ms,    4.8655ms,    4.8707ms,    4.9589ms
RunRestApi:    0.0111ms,    6.0777ms,    6.0833ms,    6.1852ms
RunRestApi:    0.0111ms,    4.0896ms,    4.0934ms,    4.1113ms
RunRestApi:    0.0069ms,    3.6266ms,    3.6315ms,    3.8100ms
RunRestApi:    0.0090ms,    4.0658ms,    4.0728ms,    4.0983ms
RunRestApi:    0.0115ms,    4.0570ms,    4.0612ms,    4.0801ms
RunRestApi:    0.0076ms,    4.1102ms,    4.1144ms,    4.1739ms
RunRestApi:    0.0094ms,    3.8520ms,    3.8562ms,    3.9503ms
RunRestApi:    0.0104ms,    4.1053ms,    4.1106ms,    4.1407ms
RunRestApi:    0.0087ms,    3.9178ms,    3.9230ms,    3.9542ms
AVG: 4.34629ms

Parallel

RunRestApi:    0.0143ms,    9.9171ms,    9.9234ms,   10.0421ms
RunRestApi:    0.0101ms,   11.1165ms,   11.1207ms,   11.6701ms
RunRestApi:    0.0083ms,   12.9034ms,   12.9076ms,   12.9230ms
RunRestApi:    0.0076ms,   15.2341ms,   15.2380ms,   15.3912ms
RunRestApi:    0.0083ms,   18.3943ms,   18.3981ms,   18.4842ms
RunRestApi:    0.0069ms,   17.4774ms,   17.4812ms,   18.0191ms
RunRestApi:    0.0171ms,   11.3870ms,   11.3912ms,   11.7083ms
RunRestApi:    0.0080ms,   21.3917ms,   21.3952ms,   22.0703ms
RunRestApi:    0.0073ms,   10.6822ms,   10.6860ms,   10.8158ms
RunRestApi:    0.0073ms,   23.7886ms,   23.7924ms,   23.8099ms
RunRestApi:    0.0087ms,   10.6685ms,   10.6724ms,   10.7557ms
RunRestApi:    0.0080ms,   26.1581ms,   26.1616ms,   26.2659ms
RunRestApi:    0.0164ms,   11.2327ms,   11.2365ms,   11.6859ms
RunRestApi:    0.0181ms,   18.3628ms,   18.3677ms,   18.7558ms
RunRestApi:    0.0087ms,   10.9292ms,   10.9334ms,   11.0738ms
RunRestApi:    0.0087ms,   14.0166ms,   14.0204ms,   14.6731ms
RunRestApi:    0.0125ms,   10.9425ms,   10.9467ms,   11.3156ms
RunRestApi:    0.0087ms,   14.2514ms,   14.2553ms,   14.4810ms
RunRestApi:    0.0101ms,   14.3120ms,   14.3169ms,   14.3351ms
RunRestApi:    0.0097ms,    3.6210ms,    3.6248ms,    3.7613ms
AVG: 14.60186ms

RexConnect

Serial

RunRexConn:    0.0020ms,    0.0111ms,    1.4684ms,    7.1539ms
RunRexConn:    0.0020ms,    0.0108ms,    1.4726ms,   10.3088ms
RunRexConn:    0.0017ms,    0.0101ms,    1.3623ms,   12.1114ms
RunRexConn:    0.0024ms,    0.0122ms,    1.2994ms,    7.0765ms
RunRexConn:    0.0020ms,    0.0115ms,    1.2738ms,    7.4356ms
RunRexConn:    0.0020ms,    0.0108ms,    1.4768ms,    7.5024ms
RunRexConn:    0.0017ms,    0.0108ms,    1.4523ms,    9.5570ms
RunRexConn:    0.0020ms,    0.0115ms,    3.1660ms,    9.5973ms
RunRexConn:    0.0020ms,    0.0108ms,    1.2644ms,    7.6190ms
RunRexConn:    0.0024ms,    0.0122ms,    1.2878ms,    7.4037ms
AVG: 8.57656ms

Parallel (edit: ran test again due to some large outlier results)

RunRexConn:    0.0041ms,    0.0230ms,    2.5543ms,   26.5861ms
RunRexConn:    0.0017ms,    0.0118ms,    3.4852ms,   36.5450ms
RunRexConn:    0.0020ms,    0.0150ms,    2.9655ms,   44.6721ms
RunRexConn:    0.0020ms,    0.0125ms,    3.8485ms,   45.6307ms
RunRexConn:    0.0020ms,    0.0129ms,    4.1942ms,   49.5768ms
RunRexConn:    0.0017ms,    0.0132ms,    4.9568ms,   50.4815ms
RunRexConn:    0.0017ms,    0.0143ms,    6.0903ms,   56.1341ms
RunRexConn:    0.0017ms,    0.0125ms,    5.0285ms,   61.8059ms
RunRexConn:    0.0020ms,    0.0118ms,    1.3616ms,   30.5589ms
RunRexConn:    0.0024ms,    0.0160ms,    1.3662ms,   42.0674ms
RunRexConn:    0.0017ms,    0.0101ms,    1.3840ms,   27.4992ms
RunRexConn:    0.0020ms,    0.0108ms,    1.5076ms,   31.4961ms
RunRexConn:    0.0017ms,    0.0101ms,    1.3522ms,   35.2375ms
RunRexConn:    0.0017ms,    0.0136ms,    6.6597ms,   84.5401ms
RunRexConn:    0.0017ms,    0.0101ms,    1.4257ms,   37.1795ms
RunRexConn:    0.0017ms,    0.0101ms,    1.2665ms,   39.4689ms
RunRexConn:    0.0017ms,    0.0104ms,    1.2574ms,   43.6776ms
RunRexConn:    0.0017ms,    0.0104ms,    1.2931ms,   56.2163ms
RunRexConn:    0.0020ms,    0.0111ms,    1.6119ms,   44.6760ms
RunRexConn:    0.0020ms,    0.0108ms,    5.2847ms,   12.9660ms
AVG: 42.850785ms

RexProClient

Serial

RunRexPro:    0.0000ms,    0.0069ms,   32.1236ms,   32.1565ms
RunRexPro:    0.0003ms,    0.0024ms,   31.7215ms,   31.7579ms
RunRexPro:    0.0003ms,    0.0024ms,   29.0628ms,   29.0950ms
RunRexPro:    0.0003ms,    0.0024ms,   29.1157ms,   29.1496ms
RunRexPro:    0.0003ms,    0.0020ms,   28.4689ms,   28.5025ms
RunRexPro:    0.0003ms,    0.0020ms,   29.6669ms,   29.7033ms
RunRexPro:    0.0003ms,    0.0031ms,   28.7461ms,   28.7783ms
RunRexPro:    0.0000ms,    0.0020ms,   33.4401ms,   33.4730ms
RunRexPro:    0.0003ms,    0.0020ms,   27.7662ms,   27.7988ms
RunRexPro:    0.0003ms,    0.0020ms,   28.4938ms,   28.5281ms
AVG: 29.8943ms

Parallel

RunRexPro:    0.0000ms,    0.0010ms,   80.7146ms,   80.7580ms
RunRexPro:    0.0003ms,    0.0017ms,   87.3377ms,   87.3643ms
RunRexPro:    0.0003ms,    0.0010ms,  119.0046ms,  119.0466ms
RunRexPro:    0.0003ms,    0.0013ms,  125.0075ms,  125.0481ms
RunRexPro:    0.0003ms,    0.0013ms,  126.9473ms,  126.9739ms
RunRexPro:    0.0003ms,    0.0013ms,  133.3236ms,  133.3537ms
RunRexPro:    0.0003ms,    0.0024ms,  134.8771ms,  134.9012ms
RunRexPro:    0.0003ms,    0.0013ms,  142.8646ms,  142.9052ms
RunRexPro:    0.0003ms,    0.0024ms,   72.4311ms,   72.4657ms
RunRexPro:    0.0003ms,    0.0017ms,   67.6030ms,   67.6271ms
RunRexPro:    0.0003ms,    0.0017ms,   83.2575ms,   83.3002ms
RunRexPro:    0.0006ms,    0.0027ms,   89.2523ms,   89.2838ms
RunRexPro:    0.0003ms,    0.0024ms,   98.3481ms,   98.3796ms
RunRexPro:    0.0003ms,    0.0017ms,  111.9634ms,  111.9956ms
RunRexPro:    0.0003ms,    0.0017ms,  122.5297ms,  122.5693ms
RunRexPro:    0.0003ms,    0.0013ms,  104.8102ms,  104.8357ms
RunRexPro:    0.0006ms,    0.0027ms,  118.1409ms,  118.1682ms
RunRexPro:    0.0000ms,    0.0010ms,  262.2349ms,  262.2605ms
RunRexPro:    0.0003ms,    0.0020ms,  110.8841ms,  110.9131ms
RunRexPro:    0.0003ms,    0.0020ms,   28.1295ms,   28.1613ms
AVG: 111.015555ms

RexProTimings3.md

Timing results when using the script g.v(4).

Rest API (Gremlin extension)

Serial

RunRestApi:    0.0090ms,    6.0980ms,    6.1019ms,    6.2121ms
RunRestApi:    0.0087ms,    7.7173ms,    7.7211ms,    7.7624ms
RunRestApi:    0.0087ms,    6.3153ms,    6.3202ms,    6.3605ms
RunRestApi:    0.0090ms,    5.9699ms,    5.9748ms,    6.1215ms
RunRestApi:    0.0097ms,    5.6277ms,    5.6326ms,    5.7183ms
RunRestApi:    0.0090ms,    6.2898ms,    6.2975ms,    6.3153ms
RunRestApi:    0.0087ms,    6.1565ms,    6.1607ms,    6.2191ms
RunRestApi:    0.0080ms,   11.2295ms,   11.2341ms,   11.3541ms
RunRestApi:    0.0090ms,    6.0749ms,    6.0788ms,    6.1201ms
RunRestApi:    0.0090ms,   12.1859ms,   12.1912ms,   12.2129ms
AVG: 7.43963ms

RexConnect

Serial

RunRexConn:    0.0020ms,    0.0125ms,    1.5156ms,   10.9296ms
RunRexConn:    0.0020ms,    0.0472ms,    1.4176ms,    8.5782ms
RunRexConn:    0.0020ms,    0.0108ms,    1.6374ms,    8.9096ms
RunRexConn:    0.0020ms,    0.0115ms,    1.5580ms,    9.0006ms
RunRexConn:    0.0017ms,    0.0136ms,    1.6728ms,   10.5681ms
RunRexConn:    0.0017ms,    0.0111ms,    1.1919ms,    9.4087ms
RunRexConn:    0.0024ms,    0.0122ms,    1.6063ms,    8.5621ms
RunRexConn:    0.0017ms,    0.0101ms,    1.5174ms,    8.7745ms
RunRexConn:    0.0020ms,    0.0122ms,    1.5975ms,   15.6971ms
RunRexConn:    0.0020ms,    0.0153ms,    1.4400ms,    9.6540ms
AVG: 10.00825ms

RexProTimings4.md

Timing results when using the script g.V.both.both.count() (which returns 16,942).

Rest API (Gremlin extension)

Serial

RunRestApi:    0.0104ms,   69.2475ms,   69.2555ms,   69.2779ms
RunRestApi:    0.0108ms,   69.1547ms,   69.1638ms,   69.3098ms
RunRestApi:    0.0139ms,   86.3116ms,   86.3196ms,   86.3445ms
RunRestApi:    0.0115ms,   71.2951ms,   71.3007ms,   71.3213ms
RunRestApi:    0.0122ms,   69.3577ms,   69.3689ms,   69.3979ms
RunRestApi:    0.0143ms,   69.5180ms,   69.5274ms,   69.5519ms
RunRestApi:    0.0115ms,   76.0301ms,   76.0367ms,   76.0605ms
RunRestApi:    0.0111ms,   69.2699ms,   69.2779ms,   69.3731ms
RunRestApi:    0.0136ms,   70.7768ms,   70.7831ms,   70.9007ms
RunRestApi:    0.0136ms,   69.1617ms,   69.1701ms,   69.4319ms
AVG: 72.09695ms

RexConnect

Serial

RunRexConn:    0.0020ms,    0.0150ms,    1.2850ms,   72.0321ms
RunRexConn:    0.0034ms,    0.0220ms,    1.5919ms,   79.5111ms
RunRexConn:    0.0031ms,    0.0213ms,    1.5534ms,   74.9028ms
RunRexConn:    0.0062ms,    0.0234ms,    1.7820ms,   72.8811ms
RunRexConn:    0.0031ms,    0.0202ms,    1.6966ms,   75.2745ms
RunRexConn:    0.0027ms,    0.0223ms,    1.7249ms,   73.7420ms
RunRexConn:    0.0027ms,    0.0188ms,    1.7263ms,   73.6132ms
RunRexConn:    0.0027ms,    0.0206ms,    2.4385ms,   76.9834ms
RunRexConn:    0.0024ms,    0.0143ms,    1.5944ms,   73.5649ms
RunRexConn:    0.0027ms,    0.0174ms,    1.7645ms,   73.6734ms
AVG: 74.61785ms

RexProTimings5.md

Timing results when using the script g.V[0..30].

Rest API (Gremlin extension)

Serial (returns JSON with ~3,300 chars)

RunRestApi:    0.0153ms,   11.2837ms,   11.2939ms,   11.3401ms
RunRestApi:    0.0178ms,   11.7922ms,   11.7996ms,   11.8346ms
RunRestApi:    0.0213ms,   11.0843ms,   11.0920ms,   11.1221ms
RunRestApi:    0.0132ms,   10.4207ms,   10.4281ms,   10.4550ms
RunRestApi:    0.0118ms,   16.6749ms,   16.6812ms,   16.7746ms
RunRestApi:    0.0132ms,   10.7788ms,   10.7840ms,   10.8141ms
RunRestApi:    0.0101ms,   10.7854ms,   10.7949ms,   10.8253ms
RunRestApi:    0.0129ms,   12.2535ms,   12.2601ms,   12.2997ms
RunRestApi:    0.0185ms,   10.8585ms,   10.8641ms,   11.1872ms
RunRestApi:    0.0139ms,   11.5158ms,   11.5196ms,   11.5623ms
AVG: 11.8215ms

RexConnect

Serial (returns JSON with ~3,800 chars)

RunRexConn:    0.0020ms,    0.0143ms,    1.7046ms,   15.2653ms
RunRexConn:    0.0017ms,    0.0111ms,    1.3599ms,   13.2677ms
RunRexConn:    0.0017ms,    0.0104ms,    1.5860ms,   13.8976ms
RunRexConn:    0.0027ms,    0.0202ms,    1.7463ms,   19.5540ms
RunRexConn:    0.0020ms,    0.0143ms,    1.6777ms,   14.7466ms
RunRexConn:    0.0034ms,    0.0230ms,    1.6371ms,   14.5279ms
RunRexConn:    0.0031ms,    0.0199ms,    1.3711ms,   12.6000ms
RunRexConn:    0.0020ms,    0.0143ms,    1.3637ms,   16.7330ms
RunRexConn:    0.0017ms,    0.0132ms,    1.1146ms,   17.3395ms
RunRexConn:    0.0020ms,    0.0118ms,    1.3424ms,   13.6439ms
AVG: 15.15755ms

RexProTimings6.md

Timing results for RexConnect 0.3.5, comparing execution of several separated requests vs. one combined request. I was curious to determine how much of an effect this would have, given that latency across a local network should be rather low.

This was performed using a RexConnectClient integration test.

Results

Query Count: 1
Separate requests: 41.7433ms total, 19ms RexConnect
Combined requests: 12.6535ms total, 8ms RexConnect
Combined improvement:3.3x (total), 2.4x (RexConnect)

Query Count: 2
Separate requests: 34.412ms total, 14ms RexConnect
Combined requests: 14.7697ms total, 10ms RexConnect
Combined improvement:2.3x (total), 1.4x (RexConnect)

Query Count: 5
Separate requests: 60.9093ms total, 27ms RexConnect
Combined requests: 23.5887ms total, 18ms RexConnect
Combined improvement:2.6x (total), 1.5x (RexConnect)

Query Count: 10
Separate requests: 130.1729ms total, 62ms RexConnect
Combined requests: 33.852ms total, 28ms RexConnect
Combined improvement:3.8x (total), 2.2x (RexConnect)

Query Count: 100
Separate requests: 1181.514ms total, 425ms RexConnect
Combined requests: 215.758ms total, 204ms RexConnect
Combined improvement:5.5x (total), 2.1x (RexConnect)

Query Count: 1000
Separate requests: 16381.4496ms total, 1906ms RexConnect
Combined requests: 808.1072ms total, 759ms RexConnect
Combined improvement:20.3x (total), 2.5x (RexConnect)

Related RexProClient issue:
dkuppitz/rexpro-client#6

Related Gremlin-Users discussion:
https://groups.google.com/forum/#!topic/gremlin-users/GFrQJfcuM_Y

New Gremlin-Users "Rest API vs. RexPro" dicussion:
https://groups.google.com/forum/#!topic/gremlin-users/e5VfFt3F0UA

Interesting considerations regarding MsgPack vs. JSON:

• https://news.ycombinator.com/item?id=4090831 -- comment regarding Javascript

  So MsgPack wins the pre-gzip size comparison by 14%, but then is nearly twenty times slower at encoding the data, and is over three times slower at decoding it.

• http://stackoverflow.com/questions/2000933/protocol-buffers-versus-json-or-bson - notably ServiceStack.Text (Fabric uses this) is very fast at JSON de/serialization