MP backpressure measurement

gistfile1.txt

* MP sibling explosion back pressure

** setup
- 4node riak, 1 riak-cs
- gof3r - riak-cs - riak in single box
  other 3 riak nodes are other 3 boxes
- 100 concurrency
- 5MB = 5242880 bytes
- 4 times / concurrency
- (* 100 4 5) 2000 MB = 2GB

export GOROOT=~/local/go
export GOPATH=~/go
export PATH=~/go/bin:~/local/go/bin:$PATH

$ dd if=/dev/urandom of=/data/static/rand.2GB bs=10M count=200

./go/bin/gof3r cp /data/static/rand.100MB s3://test/`date +%Y%m%d-%H%M%S` --no-ssl -c 5 -s 5242880

watch -n 1 'curl -s -H "Accept: text/plain" http://127.0.0.1:8088/stats | grep siblings_100'

preliminary
$ gof3r cp /data/static/rand.100MB s3://test/`date +%Y%m%d-%H%M%S` --no-ssl -c 5 -s 5242880
Duration: 1.651621444s
Throughput: (/ 100 1.65) 60.6 MB/sec
simple extrapolation => (* 20 1.65) 33.0 secs

** 1st round: release/1.5: commit 5bd77fc22a2b06d28fc56de45c7e8ad434ec947c

| concurrency | siblings_100 | duration        | notes    |
|-------------+--------------+-----------------+----------|
|           1 |            1 | 1m42.829164049s |          |
|          10 |           11 | 28.613273613s   |          |
|          20 |           19 | 28.825191807s   |          |
|          40 |           73 | 44.129474163s   |          |
|          80 |          127 | 2m40.458267737s |          |
|         100 |          101 | 1m46.184551478s |          |
|         160 |          278 | error [1]       | note [1] |
|             |              |                 |          |


** 1st round: feature/mp-backpressure: commit 3f514526957741c92a44649b6e580b56a2aa8a0d

| concurrency | siblings_100 | duration        | notes    |
|-------------+--------------+-----------------+----------|
|           1 |            1 | 1m41.526840844s |          |
|          10 |            6 | 28.283778652s   |          |
|          20 |            6 | 27.787415118s   |          |
|          40 |           13 | 34.872758661s   |          |
|          80 |          109 | 2m0.350609823s  | note [2] |
|         160 |           33 | 2m13.069422436s |          |


** 1st round: feature/mp-backpressure-simple-sleep: commit 075af4fa491e143308043e6026020b8a83183567

| concurrency | siblings_100 | duration        | notes    |
|-------------+--------------+-----------------+----------|
|           1 |            1 | 1m42.630807308s |          |
|          10 |            8 | 27.596182396s   |          |
|          20 |           13 | 27.867252402s   |          |
|          40 |           14 | 30.510393587s   |          |
|          80 |           22 | 34.36469879s    |          |
|         160 |          193 | error           | note [1] |

** 2nd round: release/1.5: commit 5bd77fc22a2b06d28fc56de45c7e8ad434ec947c

| concurrency | siblings_100 | duration        | notes |
|-------------+--------------+-----------------+-------|
|           1 |            1 | 1m42.654130343s |       |
|          10 |           10 | 27.323885895s   |       |
|          20 |           19 | 28.159426097s   |       |
|          40 |           28 | 29.751216351s   |       |
|          80 |          215 | quit            |       |

** 2nd round: feature/mp-backpressure: commit 3f514526957741c92a44649b6e580b56a2aa8a0d

| concurrency | siblings_100 | duration        | notes    |
|-------------+--------------+-----------------+----------|
|           1 |            1 | 1m42.269619607s |          |
|          10 |            8 | 28.006241234s   |          |
|          20 |            8 | 31.175024186s   |          |
|          40 |           12 | 34.475565086s   |          |
|          80 |          130 | error           | note [1] |


** 2nd round: feature/mp-backpressure-simple-sleep: commit 075af4fa491e143308043e6026020b8a83183567

| concurrency | siblings_100 | duration        | notes |
|-------------+--------------+-----------------+-------|
|           1 |            1 | 1m42.796959896s |       |
|          10 |            8 | 28.075595546s   |       |
|          20 |           21 | 29.276572076s   |       |
|          40 |           11 | 30.711816876s   |       |
|          80 |          152 | quit            |       |

** notes

note [1]: retry count over and abort itself timed out.
: 2014/08/27 11:00:06 Error aborting multipart upload: 503: ""
: gof3r error: Put http://test.s3.amazonaws.com/20140827-105753?partNumber=276&uploadId=z8pOGtPcTeCaUtn7t0LDgw%3D%3D: write tcp 127.0.0.1:8080: use of closed network connection

note [2]
- riak console.log, large obj ~ 7GB
  [warning] <0.10088.1192>@riak_kv_vnode:do_get_object:1299 Reading large object of size 6649489 
- cs console.log, no clause {error, timeout}
  [error] <0.26326.0> gen_server <0.26326.0> terminated with reason: no case clause matching {error,timeout} in riak_cs_manifest_fsm:get_and_update/6 line 316