Skip to content

Instantly share code, notes, and snippets.

@ayende
Last active February 4, 2021 11:52
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ayende/487d46e19d24285745d1073cb6bd5869 to your computer and use it in GitHub Desktop.
Save ayende/487d46e19d24285745d1073cb6bd5869 to your computer and use it in GitHub Desktop.

1 server

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 24.660ms, rate sampling interval: 10ms Thread calibration: mean lat.: 30.325ms, rate sampling interval: 10ms Thread calibration: mean lat.: 7.590ms, rate sampling interval: 10ms Thread calibration: mean lat.: 38.346ms, rate sampling interval: 13ms Thread Stats Avg Stdev Max +/- Stdev Latency 3.78ms 1.03ms 33.82ms 92.69% Req/Sec 65.85 99.81 750.00 74.07% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.65ms 75.000% 4.03ms 90.000% 4.47ms 99.000% 7.04ms 99.900% 16.77ms 99.990% 23.97ms 99.999% 33.85ms 100.000% 33.85ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.369     0.000000            2         1.00
   3.003     0.100000         4240         1.11
   3.213     0.200000         8489         1.25
   3.373     0.300000        12744         1.43
   3.519     0.400000        16977         1.67
   3.655     0.500000        21221         2.00
   3.721     0.550000        23316         2.22
   3.793     0.600000        25451         2.50
   3.867     0.650000        27533         2.86
   3.945     0.700000        29680         3.33
   4.033     0.750000        31760         4.00
   4.085     0.775000        32830         4.44
   4.143     0.800000        33912         5.00
   4.207     0.825000        34962         5.71
   4.283     0.850000        36027         6.67
   4.367     0.875000        37062         8.00
   4.415     0.887500        37596         8.89
   4.471     0.900000        38116        10.00
   4.531     0.912500        38634        11.43
   4.603     0.925000        39171        13.33
   4.699     0.937500        39715        16.00
   4.755     0.943750        39971        17.78
   4.823     0.950000        40221        20.00
   4.899     0.956250        40491        22.86
   4.999     0.962500        40755        26.67
   5.139     0.968750        41018        32.00
   5.235     0.971875        41147        35.56
   5.363     0.975000        41279        40.00
   5.527     0.978125        41410        45.71
   5.747     0.981250        41543        53.33
   6.095     0.984375        41674        64.00
   6.311     0.985938        41740        71.11
   6.571     0.987500        41809        80.00
   6.839     0.989062        41872        91.43
   7.187     0.990625        41940       106.67
   7.671     0.992188        42005       128.00
   7.951     0.992969        42038       142.22
   8.319     0.993750        42071       160.00
   8.879     0.994531        42104       182.86
   9.767     0.995313        42137       213.33
  10.631     0.996094        42170       256.00
  11.223     0.996484        42187       284.44
  11.847     0.996875        42203       320.00
  13.063     0.997266        42220       365.71
  13.791     0.997656        42236       426.67
  14.471     0.998047        42253       512.00
  14.855     0.998242        42261       568.89
  15.295     0.998437        42269       640.00
  15.879     0.998633        42278       731.43
  16.415     0.998828        42287       853.33
  16.799     0.999023        42294      1024.00
  17.039     0.999121        42298      1137.78
  17.839     0.999219        42302      1280.00
  18.239     0.999316        42307      1462.86
  19.023     0.999414        42311      1706.67
  19.695     0.999512        42315      2048.00
  20.191     0.999561        42317      2275.56
  20.639     0.999609        42319      2560.00
  21.119     0.999658        42321      2925.71
  21.727     0.999707        42323      3413.33
  21.887     0.999756        42325      4096.00
  22.063     0.999780        42326      4551.11
  22.431     0.999805        42327      5120.00
  22.575     0.999829        42328      5851.43
  23.263     0.999854        42329      6826.67
  23.679     0.999878        42330      8192.00
  23.967     0.999890        42331      9102.22
  23.967     0.999902        42331     10240.00
  25.231     0.999915        42332     11702.86
  25.231     0.999927        42332     13653.33
  27.887     0.999939        42333     16384.00
  27.887     0.999945        42333     18204.44
  27.887     0.999951        42333     20480.00
  30.015     0.999957        42334     23405.71
  30.015     0.999963        42334     27306.67
  30.015     0.999969        42334     32768.00
  30.015     0.999973        42334     36408.89
  30.015     0.999976        42334     40960.00
  33.855     0.999979        42335     46811.43
  33.855     1.000000        42335          inf

#[Mean = 3.776, StdDeviation = 1.033] #[Max = 33.824, Total count = 42335] #[Buckets = 27, SubBuckets = 2048]

44899 requests in 2.99m, 233.76MB read Requests/sec: 250.38 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 14.722ms, rate sampling interval: 19ms Thread calibration: mean lat.: 2.587ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.480ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.506ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 3.74ms 2.14ms 78.53ms 83.62% Req/Sec 65.81 98.90 555.00 72.45% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 2.89ms 75.000% 3.93ms 90.000% 7.07ms 99.000% 10.81ms 99.900% 13.77ms 99.990% 17.26ms 99.999% 78.59ms 100.000% 78.59ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.577     0.000000            1         1.00
   2.151     0.100000         4265         1.11
   2.369     0.200000         8478         1.25
   2.551     0.300000        12720         1.43
   2.715     0.400000        16966         1.67
   2.889     0.500000        21165         2.00
   2.985     0.550000        23320         2.22
   3.093     0.600000        25432         2.50
   3.247     0.650000        27547         2.86
   3.455     0.700000        29638         3.33
   3.935     0.750000        31746         4.00
   4.527     0.775000        32805         4.44
   5.119     0.800000        33864         5.00
   5.683     0.825000        34929         5.71
   6.147     0.850000        35981         6.67
   6.595     0.875000        37037         8.00
   6.827     0.887500        37566         8.89
   7.071     0.900000        38101        10.00
   7.331     0.912500        38626        11.43
   7.635     0.925000        39153        13.33
   7.979     0.937500        39686        16.00
   8.147     0.943750        39947        17.78
   8.375     0.950000        40213        20.00
   8.583     0.956250        40474        22.86
   8.815     0.962500        40740        26.67
   9.111     0.968750        41008        32.00
   9.295     0.971875        41135        35.56
   9.487     0.975000        41270        40.00
   9.711     0.978125        41406        45.71
   9.903     0.981250        41534        53.33
  10.151     0.984375        41671        64.00
  10.279     0.985938        41730        71.11
  10.479     0.987500        41796        80.00
  10.703     0.989062        41866        91.43
  10.903     0.990625        41933       106.67
  11.119     0.992188        41997       128.00
  11.295     0.992969        42028       142.22
  11.423     0.993750        42062       160.00
  11.567     0.994531        42095       182.86
  11.799     0.995313        42127       213.33
  12.055     0.996094        42160       256.00
  12.215     0.996484        42177       284.44
  12.399     0.996875        42195       320.00
  12.551     0.997266        42211       365.71
  12.695     0.997656        42226       426.67
  12.983     0.998047        42243       512.00
  13.079     0.998242        42251       568.89
  13.199     0.998437        42259       640.00
  13.495     0.998633        42268       731.43
  13.631     0.998828        42276       853.33
  13.807     0.999023        42284      1024.00
  13.895     0.999121        42288      1137.78
  14.239     0.999219        42292      1280.00
  14.655     0.999316        42297      1462.86
  14.815     0.999414        42301      1706.67
  15.319     0.999512        42305      2048.00
  15.551     0.999561        42307      2275.56
  15.735     0.999609        42309      2560.00
  15.871     0.999658        42311      2925.71
  16.199     0.999707        42313      3413.33
  16.311     0.999756        42315      4096.00
  16.343     0.999780        42316      4551.11
  16.591     0.999805        42317      5120.00
  16.799     0.999829        42318      5851.43
  16.815     0.999854        42319      6826.67
  16.911     0.999878        42320      8192.00
  17.263     0.999890        42321      9102.22
  17.263     0.999902        42321     10240.00
  18.255     0.999915        42322     11702.86
  18.255     0.999927        42322     13653.33
  20.367     0.999939        42323     16384.00
  20.367     0.999945        42323     18204.44
  20.367     0.999951        42323     20480.00
  66.879     0.999957        42324     23405.71
  66.879     0.999963        42324     27306.67
  66.879     0.999969        42324     32768.00
  66.879     0.999973        42324     36408.89
  66.879     0.999976        42324     40960.00
  78.591     0.999979        42325     46811.43
  78.591     1.000000        42325          inf

#[Mean = 3.739, StdDeviation = 2.145] #[Max = 78.528, Total count = 42325] #[Buckets = 27, SubBuckets = 2048]

44889 requests in 2.99m, 235.13MB read Requests/sec: 250.39 Transfer/sec: 1.31MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.950ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.888ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.818ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.767ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.71ms 9.14ms 716.80ms 98.92% Req/Sec 131.87 109.16 800.00 57.07% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.04ms 75.000% 4.63ms 90.000% 5.90ms 99.000% 14.66ms 99.900% 58.59ms 99.990% 454.65ms 99.999% 716.29ms 100.000% 717.31ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.355     0.000000            1         1.00
   3.255     0.100000         8489         1.11
   3.503     0.200000        16925         1.25
   3.693     0.300000        25466         1.43
   3.865     0.400000        33870         1.67
   4.037     0.500000        42363         2.00
   4.131     0.550000        46619         2.22
   4.231     0.600000        50892         2.50
   4.339     0.650000        55035         2.86
   4.463     0.700000        59329         3.33
   4.627     0.750000        63540         4.00
   4.719     0.775000        65635         4.44
   4.839     0.800000        67745         5.00
   4.979     0.825000        69852         5.71
   5.167     0.850000        71933         6.67
   5.455     0.875000        74051         8.00
   5.663     0.887500        75102         8.89
   5.899     0.900000        76155        10.00
   6.207     0.912500        77215        11.43
   6.527     0.925000        78262        13.33
   6.855     0.937500        79324        16.00
   7.031     0.943750        79858        17.78
   7.195     0.950000        80378        20.00
   7.383     0.956250        80918        22.86
   7.639     0.962500        81436        26.67
   7.959     0.968750        81965        32.00
   8.171     0.971875        82231        35.56
   8.423     0.975000        82496        40.00
   8.775     0.978125        82759        45.71
   9.327     0.981250        83021        53.33
  10.527     0.984375        83288        64.00
  11.455     0.985938        83418        71.11
  12.431     0.987500        83550        80.00
  13.719     0.989062        83682        91.43
  15.223     0.990625        83815       106.67
  16.671     0.992188        83947       128.00
  17.407     0.992969        84013       142.22
  18.223     0.993750        84079       160.00
  19.567     0.994531        84145       182.86
  21.471     0.995313        84211       213.33
  22.607     0.996094        84277       256.00
  23.343     0.996484        84312       284.44
  24.527     0.996875        84344       320.00
  25.775     0.997266        84376       365.71
  27.807     0.997656        84409       426.67
  31.471     0.998047        84442       512.00
  34.143     0.998242        84459       568.89
  36.799     0.998437        84475       640.00
  39.935     0.998633        84492       731.43
  47.327     0.998828        84508       853.33
  68.607     0.999023        84525      1024.00
  78.783     0.999121        84533      1137.78
 103.167     0.999219        84541      1280.00
 120.575     0.999316        84550      1462.86
 163.455     0.999414        84558      1706.67
 211.327     0.999512        84566      2048.00
 227.455     0.999561        84570      2275.56
 237.823     0.999609        84574      2560.00
 243.455     0.999658        84579      2925.71
 265.983     0.999707        84583      3413.33
 288.255     0.999756        84587      4096.00
 309.503     0.999780        84589      4551.11
 320.255     0.999805        84591      5120.00
 361.215     0.999829        84593      5851.43
 372.991     0.999854        84595      6826.67
 447.999     0.999878        84597      8192.00
 449.535     0.999890        84598      9102.22
 454.655     0.999902        84599     10240.00
 464.383     0.999915        84600     11702.86
 466.175     0.999927        84601     13653.33
 470.783     0.999939        84602     16384.00
 700.415     0.999945        84603     18204.44
 700.415     0.999951        84603     20480.00
 700.927     0.999957        84604     23405.71
 700.927     0.999963        84604     27306.67
 704.511     0.999969        84605     32768.00
 704.511     0.999973        84605     36408.89
 704.511     0.999976        84605     40960.00
 716.287     0.999979        84606     46811.43
 716.287     0.999982        84606     54613.33
 716.287     0.999985        84606     65536.00
 716.287     0.999986        84606     72817.78
 716.287     0.999988        84606     81920.00
 717.311     0.999989        84607     93622.86
 717.311     1.000000        84607          inf

#[Mean = 4.715, StdDeviation = 9.141] #[Max = 716.800, Total count = 84607] #[Buckets = 27, SubBuckets = 2048]

89732 requests in 2.98m, 469.59MB read Requests/sec: 501.04 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.468ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.483ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.461ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.493ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 5.39ms 27.01ms 666.11ms 99.37% Req/Sec 131.86 111.69 2.67k 56.99% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 2.99ms 75.000% 3.64ms 90.000% 6.16ms 99.000% 12.63ms 99.900% 496.38ms 99.990% 627.71ms 99.999% 652.80ms 100.000% 666.62ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.498     0.000000            1         1.00
   2.227     0.100000         8529         1.11
   2.467     0.200000        16986         1.25
   2.653     0.300000        25450         1.43
   2.819     0.400000        33928         1.67
   2.993     0.500000        42349         2.00
   3.091     0.550000        46594         2.22
   3.197     0.600000        50815         2.50
   3.315     0.650000        55049         2.86
   3.455     0.700000        59264         3.33
   3.639     0.750000        63496         4.00
   3.755     0.775000        65604         4.44
   3.897     0.800000        67713         5.00
   4.115     0.825000        69834         5.71
   4.511     0.850000        71936         6.67
   5.255     0.875000        74044         8.00
   5.707     0.887500        75102         8.89
   6.155     0.900000        76161        10.00
   6.611     0.912500        77223        11.43
   7.027     0.925000        78284        13.33
   7.427     0.937500        79335        16.00
   7.643     0.943750        79870        17.78
   7.867     0.950000        80395        20.00
   8.139     0.956250        80923        22.86
   8.463     0.962500        81453        26.67
   8.855     0.968750        81978        32.00
   9.103     0.971875        82247        35.56
   9.383     0.975000        82507        40.00
   9.711     0.978125        82770        45.71
  10.087     0.981250        83035        53.33
  10.711     0.984375        83299        64.00
  11.119     0.985938        83436        71.11
  11.559     0.987500        83564        80.00
  12.127     0.989062        83696        91.43
  13.119     0.990625        83828       106.67
  16.047     0.992188        83960       128.00
  19.135     0.992969        84027       142.22
  36.735     0.993750        84093       160.00
 112.639     0.994531        84159       182.86
 144.127     0.995313        84225       213.33
 189.951     0.996094        84291       256.00
 214.143     0.996484        84324       284.44
 245.119     0.996875        84357       320.00
 334.847     0.997266        84391       365.71
 364.543     0.997656        84423       426.67
 388.863     0.998047        84456       512.00
 401.151     0.998242        84473       568.89
 435.967     0.998437        84489       640.00
 454.399     0.998633        84506       731.43
 479.743     0.998828        84522       853.33
 498.943     0.999023        84539      1024.00
 508.927     0.999121        84547      1137.78
 542.207     0.999219        84555      1280.00
 567.295     0.999316        84564      1462.86
 577.023     0.999414        84572      1706.67
 588.799     0.999512        84580      2048.00
 593.919     0.999561        84584      2275.56
 600.063     0.999609        84588      2560.00
 607.231     0.999658        84593      2925.71
 609.791     0.999707        84597      3413.33
 614.399     0.999756        84601      4096.00
 616.959     0.999780        84603      4551.11
 618.495     0.999805        84605      5120.00
 621.567     0.999829        84607      5851.43
 623.615     0.999854        84609      6826.67
 625.663     0.999878        84611      8192.00
 626.175     0.999890        84612      9102.22
 627.711     0.999902        84613     10240.00
 630.271     0.999915        84614     11702.86
 631.295     0.999927        84615     13653.33
 632.319     0.999939        84616     16384.00
 633.343     0.999945        84617     18204.44
 633.343     0.999951        84617     20480.00
 647.679     0.999957        84618     23405.71
 647.679     0.999963        84618     27306.67
 652.287     0.999969        84619     32768.00
 652.287     0.999973        84619     36408.89
 652.287     0.999976        84619     40960.00
 652.799     0.999979        84620     46811.43
 652.799     0.999982        84620     54613.33
 652.799     0.999985        84620     65536.00
 652.799     0.999986        84620     72817.78
 652.799     0.999988        84620     81920.00
 666.623     0.999989        84621     93622.86
 666.623     1.000000        84621          inf

#[Mean = 5.387, StdDeviation = 27.012] #[Max = 666.112, Total count = 84621] #[Buckets = 27, SubBuckets = 2048]

89745 requests in 2.98m, 470.11MB read Requests/sec: 501.13 Transfer/sec: 2.63MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.356ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.532ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.551ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.523ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.35ms 19.49ms 718.34ms 99.53% Req/Sec 197.80 79.99 6.11k 73.09% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 2.94ms 75.000% 3.45ms 90.000% 4.30ms 99.000% 9.48ms 99.900% 390.91ms 99.990% 543.23ms 99.999% 664.58ms 100.000% 718.85ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.489     0.000000            1         1.00
   2.219     0.100000        12697         1.11
   2.449     0.200000        25389         1.25
   2.627     0.300000        38105         1.43
   2.785     0.400000        50842         1.67
   2.943     0.500000        63562         2.00
   3.025     0.550000        69828         2.22
   3.115     0.600000        76177         2.50
   3.213     0.650000        82627         2.86
   3.321     0.700000        88904         3.33
   3.451     0.750000        95202         4.00
   3.527     0.775000        98379         4.44
   3.613     0.800000       101588         5.00
   3.713     0.825000       104722         5.71
   3.841     0.850000       107912         6.67
   4.013     0.875000       111060         8.00
   4.135     0.887500       112656         8.89
   4.299     0.900000       114246        10.00
   4.547     0.912500       115825        11.43
   4.939     0.925000       117405        13.33
   5.415     0.937500       118993        16.00
   5.707     0.943750       119783        17.78
   5.983     0.950000       120582        20.00
   6.287     0.956250       121373        22.86
   6.583     0.962500       122164        26.67
   6.927     0.968750       122961        32.00
   7.103     0.971875       123353        35.56
   7.323     0.975000       123757        40.00
   7.567     0.978125       124150        45.71
   7.919     0.981250       124544        53.33
   8.303     0.984375       124941        64.00
   8.575     0.985938       125140        71.11
   8.863     0.987500       125338        80.00
   9.175     0.989062       125534        91.43
   9.711     0.990625       125735       106.67
  10.567     0.992188       125931       128.00
  11.183     0.992969       126031       142.22
  12.095     0.993750       126129       160.00
  13.959     0.994531       126228       182.86
  22.799     0.995313       126328       213.33
  79.743     0.996094       126427       256.00
 106.687     0.996484       126476       284.44
 142.079     0.996875       126526       320.00
 182.655     0.997266       126575       365.71
 222.079     0.997656       126625       426.67
 263.423     0.998047       126675       512.00
 286.719     0.998242       126700       568.89
 316.927     0.998437       126724       640.00
 343.295     0.998633       126749       731.43
 371.199     0.998828       126775       853.33
 397.055     0.999023       126799      1024.00
 403.199     0.999121       126812      1137.78
 415.231     0.999219       126824      1280.00
 428.031     0.999316       126836      1462.86
 442.367     0.999414       126848      1706.67
 452.863     0.999512       126861      2048.00
 462.335     0.999561       126867      2275.56
 473.855     0.999609       126873      2560.00
 483.583     0.999658       126880      2925.71
 493.567     0.999707       126885      3413.33
 506.111     0.999756       126892      4096.00
 514.047     0.999780       126895      4551.11
 522.495     0.999805       126898      5120.00
 527.871     0.999829       126901      5851.43
 533.503     0.999854       126904      6826.67
 538.623     0.999878       126907      8192.00
 543.231     0.999890       126909      9102.22
 549.375     0.999902       126910     10240.00
 553.471     0.999915       126912     11702.86
 558.591     0.999927       126914     13653.33
 564.223     0.999939       126915     16384.00
 565.247     0.999945       126916     18204.44
 565.247     0.999951       126916     20480.00
 588.799     0.999957       126917     23405.71
 603.647     0.999963       126918     27306.67
 607.743     0.999969       126919     32768.00
 607.743     0.999973       126919     36408.89
 607.743     0.999976       126919     40960.00
 612.863     0.999979       126920     46811.43
 612.863     0.999982       126920     54613.33
 664.575     0.999985       126921     65536.00
 664.575     0.999986       126921     72817.78
 664.575     0.999988       126921     81920.00
 664.575     0.999989       126921     93622.86
 664.575     0.999991       126921    109226.67
 718.847     0.999992       126922    131072.00
 718.847     1.000000       126922          inf

#[Mean = 4.347, StdDeviation = 19.485] #[Max = 718.336, Total count = 126922] #[Buckets = 27, SubBuckets = 2048]

134550 requests in 2.98m, 704.55MB read Requests/sec: 751.42 Transfer/sec: 3.93MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 786.737ms, rate sampling interval: 5152ms Thread calibration: mean lat.: 821.466ms, rate sampling interval: 5615ms Thread calibration: mean lat.: 628.782ms, rate sampling interval: 4464ms Thread calibration: mean lat.: 557.545ms, rate sampling interval: 4128ms Thread Stats Avg Stdev Max +/- Stdev Latency 7.22ms 31.14ms 1.01s 99.34% Req/Sec 187.03 0.24 189.00 99.28% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.41ms 75.000% 5.64ms 90.000% 8.38ms 99.000% 25.30ms 99.900% 630.78ms 99.990% 936.96ms 99.999% 1.01s 100.000% 1.01s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.391     0.000000            1         1.00
   3.443     0.100000        12724         1.11
   3.721     0.200000        25407         1.25
   3.949     0.300000        38180         1.43
   4.167     0.400000        50883         1.67
   4.411     0.500000        63593         2.00
   4.555     0.550000        69990         2.22
   4.719     0.600000        76297         2.50
   4.927     0.650000        82637         2.86
   5.207     0.700000        88909         3.33
   5.643     0.750000        95274         4.00
   5.927     0.775000        98425         4.44
   6.247     0.800000       101607         5.00
   6.563     0.825000       104783         5.71
   6.919     0.850000       107957         6.67
   7.399     0.875000       111132         8.00
   7.779     0.887500       112715         8.89
   8.383     0.900000       114302        10.00
   9.327     0.912500       115883        11.43
  10.431     0.925000       117463        13.33
  11.583     0.937500       119054        16.00
  12.111     0.943750       119854        17.78
  12.647     0.950000       120639        20.00
  13.231     0.956250       121433        22.86
  13.863     0.962500       122233        26.67
  14.623     0.968750       123020        32.00
  15.119     0.971875       123420        35.56
  15.743     0.975000       123818        40.00
  16.543     0.978125       124210        45.71
  17.391     0.981250       124608        53.33
  18.639     0.984375       125004        64.00
  19.631     0.985938       125202        71.11
  21.375     0.987500       125401        80.00
  23.407     0.989062       125599        91.43
  27.071     0.990625       125797       106.67
  32.655     0.992188       125995       128.00
  35.807     0.992969       126096       142.22
  40.831     0.993750       126194       160.00
  52.511     0.994531       126293       182.86
  95.615     0.995313       126392       213.33
 131.711     0.996094       126491       256.00
 150.527     0.996484       126541       284.44
 182.399     0.996875       126592       320.00
 219.007     0.997266       126640       365.71
 264.447     0.997656       126690       426.67
 323.583     0.998047       126739       512.00
 379.391     0.998242       126764       568.89
 438.527     0.998437       126789       640.00
 508.415     0.998633       126815       731.43
 575.999     0.998828       126839       853.33
 637.951     0.999023       126863      1024.00
 671.743     0.999121       126876      1137.78
 702.463     0.999219       126888      1280.00
 744.447     0.999316       126901      1462.86
 770.047     0.999414       126913      1706.67
 807.423     0.999512       126925      2048.00
 826.367     0.999561       126932      2275.56
 837.631     0.999609       126938      2560.00
 856.063     0.999658       126944      2925.71
 870.399     0.999707       126950      3413.33
 881.151     0.999756       126956      4096.00
 892.927     0.999780       126960      4551.11
 904.703     0.999805       126963      5120.00
 915.967     0.999829       126966      5851.43
 922.623     0.999854       126970      6826.67
 932.351     0.999878       126973      8192.00
 936.959     0.999890       126974      9102.22
 942.591     0.999902       126975     10240.00
 952.319     0.999915       126977     11702.86
 956.927     0.999927       126978     13653.33
 977.919     0.999939       126980     16384.00
 984.063     0.999945       126981     18204.44
 984.063     0.999951       126981     20480.00
 988.159     0.999957       126982     23405.71
 993.791     0.999963       126983     27306.67
 998.399     0.999969       126984     32768.00
 998.399     0.999973       126984     36408.89
 998.399     0.999976       126984     40960.00
1003.519     0.999979       126985     46811.43
1003.519     0.999982       126985     54613.33
1008.639     0.999985       126986     65536.00
1008.639     0.999986       126986     72817.78
1008.639     0.999988       126986     81920.00
1008.639     0.999989       126986     93622.86
1008.639     0.999991       126986    109226.67
1014.783     0.999992       126987    131072.00
1014.783     1.000000       126987          inf

#[Mean = 7.223, StdDeviation = 31.138] #[Max = 1014.272, Total count = 126987] #[Buckets = 27, SubBuckets = 2048]

134602 requests in 2.99m, 718.44MB read Socket errors: connect 0, read 0, write 0, timeout 128 Requests/sec: 751.18 Transfer/sec: 4.01MB

3 servers

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.751ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.790ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.706ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.757ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.98ms 2.48ms 104.51ms 93.26% Req/Sec 197.68 72.22 777.00 65.90% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.34ms 75.000% 5.25ms 90.000% 6.78ms 99.000% 15.72ms 99.900% 24.32ms 99.990% 62.27ms 99.999% 88.32ms 100.000% 104.57ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.425     0.000000            1         1.00
   3.427     0.100000        12757         1.11
   3.695     0.200000        25405         1.25
   3.915     0.300000        38192         1.43
   4.123     0.400000        51017         1.67
   4.339     0.500000        63528         2.00
   4.467     0.550000        69983         2.22
   4.607     0.600000        76213         2.50
   4.775     0.650000        82648         2.86
   4.983     0.700000        88936         3.33
   5.251     0.750000        95210         4.00
   5.423     0.775000        98443         4.44
   5.623     0.800000       101565         5.00
   5.863     0.825000       104779         5.71
   6.135     0.850000       107913         6.67
   6.431     0.875000       111096         8.00
   6.595     0.887500       112670         8.89
   6.783     0.900000       114287        10.00
   6.995     0.912500       115863        11.43
   7.259     0.925000       117430        13.33
   7.615     0.937500       119015        16.00
   7.891     0.943750       119805        17.78
   8.335     0.950000       120603        20.00
   9.103     0.956250       121397        22.86
  10.287     0.962500       122187        26.67
  11.471     0.968750       122983        32.00
  11.975     0.971875       123381        35.56
  12.439     0.975000       123775        40.00
  12.975     0.978125       124179        45.71
  13.511     0.981250       124568        53.33
  14.103     0.984375       124964        64.00
  14.471     0.985938       125166        71.11
  14.839     0.987500       125359        80.00
  15.375     0.989062       125561        91.43
  15.935     0.990625       125757       106.67
  16.703     0.992188       125958       128.00
  17.007     0.992969       126053       142.22
  17.407     0.993750       126154       160.00
  17.759     0.994531       126252       182.86
  18.127     0.995313       126350       213.33
  18.751     0.996094       126452       256.00
  19.023     0.996484       126499       284.44
  19.535     0.996875       126550       320.00
  19.999     0.997266       126600       365.71
  20.703     0.997656       126648       426.67
  21.583     0.998047       126698       512.00
  22.031     0.998242       126722       568.89
  22.367     0.998437       126747       640.00
  22.847     0.998633       126772       731.43
  23.391     0.998828       126797       853.33
  24.495     0.999023       126822      1024.00
  25.135     0.999121       126835      1137.78
  26.079     0.999219       126846      1280.00
  26.927     0.999316       126859      1462.86
  28.399     0.999414       126871      1706.67
  30.351     0.999512       126884      2048.00
  30.911     0.999561       126890      2275.56
  32.063     0.999609       126896      2560.00
  33.503     0.999658       126902      2925.71
  35.391     0.999707       126908      3413.33
  37.279     0.999756       126915      4096.00
  39.839     0.999780       126918      4551.11
  41.695     0.999805       126921      5120.00
  44.127     0.999829       126924      5851.43
  48.895     0.999854       126927      6826.67
  56.511     0.999878       126930      8192.00
  62.271     0.999890       126932      9102.22
  63.647     0.999902       126933     10240.00
  71.231     0.999915       126935     11702.86
  73.023     0.999927       126936     13653.33
  78.335     0.999939       126938     16384.00
  78.847     0.999945       126939     18204.44
  78.847     0.999951       126939     20480.00
  79.359     0.999957       126940     23405.71
  84.543     0.999963       126941     27306.67
  86.335     0.999969       126942     32768.00
  86.335     0.999973       126942     36408.89
  86.335     0.999976       126942     40960.00
  87.999     0.999979       126943     46811.43
  87.999     0.999982       126943     54613.33
  88.319     0.999985       126944     65536.00
  88.319     0.999986       126944     72817.78
  88.319     0.999988       126944     81920.00
  88.319     0.999989       126944     93622.86
  88.319     0.999991       126944    109226.67
 104.575     0.999992       126945    131072.00

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.984ms, rate sampling interval: 10ms Thread calibration: mean lat.: 5.008ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.229ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.778ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 34.22ms 220.94ms 2.88s 98.06% Req/Sec 196.01 99.01 6.78k 78.31% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.62ms 75.000% 7.66ms 90.000% 13.48ms 99.000% 1.35s 99.900% 2.66s 99.990% 2.82s 99.999% 2.87s 100.000% 2.88s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.405     0.000000            1         1.00
   3.463     0.100000        12757         1.11
   3.769     0.200000        25388         1.25
   4.029     0.300000        38190         1.43
   4.295     0.400000        50922         1.67
   4.623     0.500000        63611         2.00
   4.847     0.550000        69858         2.22
   5.187     0.600000        76205         2.50
   5.743     0.650000        82513         2.86
   6.627     0.700000        88854         3.33
   7.655     0.750000        95211         4.00
   8.535     0.775000        98377         4.44
   9.615     0.800000       101548         5.00
  10.703     0.825000       104730         5.71
  11.695     0.850000       107896         6.67
  12.575     0.875000       111067         8.00
  13.015     0.887500       112648         8.89
  13.479     0.900000       114243        10.00
  14.023     0.912500       115822        11.43
  14.719     0.925000       117421        13.33
  15.703     0.937500       118996        16.00
  16.431     0.943750       119792        17.78
  17.071     0.950000       120583        20.00
  17.759     0.956250       121375        22.86
  18.703     0.962500       122169        26.67
  21.359     0.968750       122962        32.00
  24.495     0.971875       123358        35.56
  40.383     0.975000       123755        40.00
 119.743     0.978125       124151        45.71
 307.199     0.981250       124548        53.33
 554.495     0.984375       124947        64.00
 764.415     0.985938       125143        71.11
 995.839     0.987500       125341        80.00
1212.415     0.989062       125539        91.43
1434.623     0.990625       125738       106.67
1646.591     0.992188       125936       128.00
1752.063     0.992969       126036       142.22
1861.631     0.993750       126134       160.00
1975.295     0.994531       126233       182.86
2085.887     0.995313       126335       213.33
2189.311     0.996094       126433       256.00
2256.895     0.996484       126481       284.44
2299.903     0.996875       126532       320.00
2365.439     0.997266       126580       365.71
2418.687     0.997656       126631       426.67
2488.319     0.998047       126681       512.00
2516.991     0.998242       126705       568.89
2543.615     0.998437       126730       640.00
2580.479     0.998633       126754       731.43
2623.487     0.998828       126779       853.33
2664.447     0.999023       126804      1024.00
2674.687     0.999121       126817      1137.78
2686.975     0.999219       126828      1280.00
2715.647     0.999316       126841      1462.86
2734.079     0.999414       126854      1706.67
2760.703     0.999512       126868      2048.00
2764.799     0.999561       126872      2275.56
2772.991     0.999609       126879      2560.00
2781.183     0.999658       126884      2925.71
2791.423     0.999707       126893      3413.33
2799.615     0.999756       126898      4096.00
2803.711     0.999780       126902      4551.11
2805.759     0.999805       126904      5120.00
2807.807     0.999829       126906      5851.43
2815.999     0.999854       126910      6826.67
2822.143     0.999878       126914      8192.00
2822.143     0.999890       126914      9102.22
2826.239     0.999902       126915     10240.00
2832.383     0.999915       126917     11702.86
2840.575     0.999927       126918     13653.33
2844.671     0.999939       126920     16384.00
2846.719     0.999945       126921     18204.44
2846.719     0.999951       126921     20480.00
2850.815     0.999957       126922     23405.71
2854.911     0.999963       126924     27306.67
2854.911     0.999969       126924     32768.00
2854.911     0.999973       126924     36408.89
2854.911     0.999976       126924     40960.00
2856.959     0.999979       126925     46811.43
2856.959     0.999982       126925     54613.33
2871.295     0.999985       126926     65536.00
2871.295     0.999986       126926     72817.78
2871.295     0.999988       126926     81920.00
2871.295     0.999989       126926     93622.86
2871.295     0.999991       126926    109226.67
2877.439     0.999992       126927    131072.00
2877.439     1.000000       126927          inf

#[Mean = 34.219, StdDeviation = 220.942] #[Max = 2875.392, Total count = 126927] #[Buckets = 27, SubBuckets = 2048]

134555 requests in 2.98m, 718.18MB read Socket errors: connect 0, read 0, write 0, timeout 47 Requests/sec: 751.40 Transfer/sec: 4.01MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 450.770ms, rate sampling interval: 3952ms Thread calibration: mean lat.: 383.596ms, rate sampling interval: 3473ms Thread calibration: mean lat.: 338.162ms, rate sampling interval: 3155ms Thread calibration: mean lat.: 290.105ms, rate sampling interval: 2779ms ^C Thread Stats Avg Stdev Max +/- Stdev Latency 7.10ms 29.45ms 594.94ms 98.62% Req/Sec 186.94 1.00 191.00 91.86% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.01ms 75.000% 4.46ms 90.000% 4.96ms 99.000% 112.64ms 99.900% 441.86ms 99.990% 544.26ms 99.999% 584.19ms 100.000% 595.46ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.367     0.000000            1         1.00
   3.287     0.100000         5490         1.11
   3.521     0.200000        10964         1.25
   3.699     0.300000        16427         1.43
   3.857     0.400000        21909         1.67
   4.007     0.500000        27382         2.00
   4.087     0.550000        30116         2.22
   4.171     0.600000        32851         2.50
   4.259     0.650000        35597         2.86
   4.355     0.700000        38326         3.33
   4.463     0.750000        41101         4.00
   4.519     0.775000        42445         4.44
   4.583     0.800000        43830         5.00
   4.655     0.825000        45207         5.71
   4.735     0.850000        46597         6.67
   4.831     0.875000        47913         8.00
   4.895     0.887500        48614         8.89
   4.963     0.900000        49289        10.00
   5.051     0.912500        49979        11.43
   5.167     0.925000        50660        13.33
   5.315     0.937500        51338        16.00
   5.415     0.943750        51683        17.78
   5.547     0.950000        52020        20.00
   5.703     0.956250        52358        22.86
   5.947     0.962500        52699        26.67
   6.411     0.968750        53042        32.00
   6.899     0.971875        53214        35.56
   7.767     0.975000        53385        40.00
   9.015     0.978125        53554        45.71
  11.639     0.981250        53725        53.33
  20.511     0.984375        53896        64.00
  31.263     0.985938        53982        71.11
  57.471     0.987500        54067        80.00
  92.095     0.989062        54154        91.43
 127.103     0.990625        54238       106.67
 156.543     0.992188        54324       128.00
 186.751     0.992969        54367       142.22
 216.319     0.993750        54409       160.00
 245.375     0.994531        54452       182.86
 272.895     0.995313        54496       213.33
 298.495     0.996094        54540       256.00
 309.247     0.996484        54559       284.44
 319.487     0.996875        54580       320.00
 343.039     0.997266        54602       365.71
 362.495     0.997656        54623       426.67
 387.327     0.998047        54645       512.00
 399.359     0.998242        54655       568.89
 413.439     0.998437        54667       640.00
 419.327     0.998633        54677       731.43
 430.591     0.998828        54687       853.33
 446.463     0.999023        54698      1024.00
 456.703     0.999121        54703      1137.78
 463.359     0.999219        54710      1280.00
 467.199     0.999316        54714      1462.86
 471.295     0.999414        54720      1706.67
 474.879     0.999512        54726      2048.00
 478.975     0.999561        54727      2275.56
 483.583     0.999609        54730      2560.00
 489.215     0.999658        54733      2925.71
 494.079     0.999707        54735      3413.33
 498.431     0.999756        54738      4096.00
 500.223     0.999780        54739      4551.11
 507.903     0.999805        54741      5120.00
 508.415     0.999829        54742      5851.43
 514.559     0.999854        54743      6826.67
 525.311     0.999878        54745      8192.00
 525.311     0.999890        54745      9102.22
 544.255     0.999902        54747     10240.00
 544.255     0.999915        54747     11702.86
 544.255     0.999927        54747     13653.33
 581.119     0.999939        54748     16384.00
 581.119     0.999945        54748     18204.44
 582.143     0.999951        54749     20480.00
 582.143     0.999957        54749     23405.71
 582.143     0.999963        54749     27306.67
 584.191     0.999969        54750     32768.00
 584.191     0.999973        54750     36408.89
 584.191     0.999976        54750     40960.00
 584.191     0.999979        54750     46811.43
 584.191     0.999982        54750     54613.33
 595.455     0.999985        54751     65536.00
 595.455     1.000000        54751          inf

#[Mean = 7.105, StdDeviation = 29.445] #[Max = 594.944, Total count = 54751] #[Buckets = 27, SubBuckets = 2048]

62375 requests in 1.40m, 325.02MB read Socket errors: connect 0, read 0, write 0, timeout 118 Requests/sec: 744.46 Transfer/sec: 3.88MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.749ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.881ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.842ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.770ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 5.10ms 11.72ms 805.89ms 99.35% Req/Sec 131.73 110.16 1.64k 55.71% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.16ms 75.000% 4.86ms 90.000% 6.58ms 99.000% 14.11ms 99.900% 183.17ms 99.990% 482.05ms 99.999% 553.47ms 100.000% 806.40ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.399     0.000000            1         1.00
   3.321     0.100000         8476         1.11
   3.585     0.200000        16943         1.25
   3.791     0.300000        25485         1.43
   3.971     0.400000        33936         1.67
   4.163     0.500000        42372         2.00
   4.271     0.550000        46679         2.22
   4.383     0.600000        50799         2.50
   4.511     0.650000        55134         2.86
   4.655     0.700000        59301         3.33
   4.859     0.750000        63537         4.00
   4.983     0.775000        65626         4.44
   5.143     0.800000        67762         5.00
   5.355     0.825000        69836         5.71
   5.671     0.850000        71965         6.67
   6.075     0.875000        74070         8.00
   6.335     0.887500        75125         8.89
   6.583     0.900000        76191        10.00
   6.859     0.912500        77239        11.43
   7.159     0.925000        78302        13.33
   7.515     0.937500        79356        16.00
   7.783     0.943750        79887        17.78
   8.099     0.950000        80413        20.00
   8.463     0.956250        80944        22.86
   8.895     0.962500        81478        26.67
   9.447     0.968750        82002        32.00
   9.823     0.971875        82265        35.56
  10.287     0.975000        82531        40.00
  10.911     0.978125        82795        45.71
  11.583     0.981250        83061        53.33
  12.255     0.984375        83323        64.00
  12.679     0.985938        83457        71.11
  13.207     0.987500        83589        80.00
  13.815     0.989062        83722        91.43
  14.415     0.990625        83852       106.67
  15.551     0.992188        83985       128.00
  16.375     0.992969        84052       142.22
  17.055     0.993750        84117       160.00
  17.727     0.994531        84183       182.86
  18.543     0.995313        84250       213.33
  20.239     0.996094        84315       256.00
  21.311     0.996484        84348       284.44
  22.799     0.996875        84381       320.00
  25.183     0.997266        84414       365.71
  31.551     0.997656        84447       426.67
  43.519     0.998047        84480       512.00
  50.335     0.998242        84497       568.89
  66.943     0.998437        84513       640.00
  96.895     0.998633        84530       731.43
 118.143     0.998828        84546       853.33
 189.311     0.999023        84563      1024.00
 239.487     0.999121        84571      1137.78
 258.943     0.999219        84579      1280.00
 278.527     0.999316        84588      1462.86
 289.279     0.999414        84596      1706.67
 310.271     0.999512        84604      2048.00
 323.327     0.999561        84608      2275.56
 332.543     0.999609        84612      2560.00
 365.823     0.999658        84617      2925.71
 412.415     0.999707        84621      3413.33
 436.735     0.999756        84625      4096.00
 441.599     0.999780        84627      4551.11
 448.511     0.999805        84629      5120.00
 450.815     0.999829        84631      5851.43
 452.095     0.999854        84633      6826.67
 464.383     0.999878        84635      8192.00
 474.111     0.999890        84636      9102.22
 482.047     0.999902        84637     10240.00
 485.887     0.999915        84638     11702.86
 503.807     0.999927        84639     13653.33
 506.879     0.999939        84640     16384.00
 537.087     0.999945        84641     18204.44
 537.087     0.999951        84641     20480.00
 539.135     0.999957        84642     23405.71
 539.135     0.999963        84642     27306.67
 549.375     0.999969        84643     32768.00
 549.375     0.999973        84643     36408.89
 549.375     0.999976        84643     40960.00
 553.471     0.999979        84644     46811.43
 553.471     0.999982        84644     54613.33
 553.471     0.999985        84644     65536.00
 553.471     0.999986        84644     72817.78
 553.471     0.999988        84644     81920.00
 806.399     0.999989        84645     93622.86
 806.399     1.000000        84645          inf

#[Mean = 5.098, StdDeviation = 11.717] #[Max = 805.888, Total count = 84645] #[Buckets = 27, SubBuckets = 2048]

89769 requests in 2.99m, 469.79MB read Requests/sec: 500.93 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.861ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.791ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.808ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.762ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 29.58ms 202.84ms 3.02s 98.31% Req/Sec 131.48 125.90 3.60k 50.66% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.58ms 75.000% 7.48ms 90.000% 11.76ms 99.000% 1.05s 99.900% 2.64s 99.990% 2.84s 99.999% 2.91s 100.000% 3.02s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.339     0.000000            1         1.00
   3.391     0.100000         8484         1.11
   3.699     0.200000        16957         1.25
   3.967     0.300000        25432         1.43
   4.239     0.400000        33937         1.67
   4.583     0.500000        42363         2.00
   4.811     0.550000        46562         2.22
   5.147     0.600000        50796         2.50
   5.715     0.650000        55025         2.86
   6.631     0.700000        59249         3.33
   7.483     0.750000        63485         4.00
   7.959     0.775000        65600         4.44
   8.487     0.800000        67716         5.00
   9.135     0.825000        69830         5.71
   9.823     0.850000        71939         6.67
  10.711     0.875000        74054         8.00
  11.255     0.887500        75115         8.89
  11.759     0.900000        76183        10.00
  12.239     0.912500        77235        11.43
  12.767     0.925000        78288        13.33
  13.463     0.937500        79357        16.00
  13.895     0.943750        79872        17.78
  14.495     0.950000        80401        20.00
  15.399     0.956250        80931        22.86
  16.799     0.962500        81462        26.67
  18.063     0.968750        81991        32.00
  19.103     0.971875        82255        35.56
  22.095     0.975000        82518        40.00
  38.719     0.978125        82781        45.71
 158.719     0.981250        83046        53.33
 318.463     0.984375        83310        64.00
 456.959     0.985938        83442        71.11
 671.743     0.987500        83575        80.00
 923.135     0.989062        83708        91.43
1177.599     0.990625        83840       106.67
1426.431     0.992188        83971       128.00
1537.023     0.992969        84037       142.22
1680.383     0.993750        84105       160.00
1814.527     0.994531        84170       182.86
1943.551     0.995313        84237       213.33
2105.343     0.996094        84302       256.00
2156.543     0.996484        84335       284.44
2224.127     0.996875        84368       320.00
2310.143     0.997266        84401       365.71
2383.871     0.997656        84435       426.67
2437.119     0.998047        84467       512.00
2469.887     0.998242        84484       568.89
2504.703     0.998437        84500       640.00
2553.855     0.998633        84517       731.43
2592.767     0.998828        84534       853.33
2639.871     0.999023        84550      1024.00
2670.591     0.999121        84558      1137.78
2678.783     0.999219        84566      1280.00
2699.263     0.999316        84575      1462.86
2713.599     0.999414        84583      1706.67
2727.935     0.999512        84592      2048.00
2732.031     0.999561        84595      2275.56
2746.367     0.999609        84599      2560.00
2764.799     0.999658        84604      2925.71
2777.087     0.999707        84608      3413.33
2789.375     0.999756        84612      4096.00
2793.471     0.999780        84614      4551.11
2801.663     0.999805        84616      5120.00
2811.903     0.999829        84619      5851.43
2815.999     0.999854        84620      6826.67
2826.239     0.999878        84622      8192.00
2836.479     0.999890        84623      9102.22
2844.671     0.999902        84624     10240.00
2856.959     0.999915        84625     11702.86
2861.055     0.999927        84627     13653.33
2861.055     0.999939        84627     16384.00
2873.343     0.999945        84628     18204.44
2873.343     0.999951        84628     20480.00
2875.391     0.999957        84629     23405.71
2875.391     0.999963        84629     27306.67
2887.679     0.999969        84630     32768.00
2887.679     0.999973        84630     36408.89
2887.679     0.999976        84630     40960.00
2910.207     0.999979        84631     46811.43
2910.207     0.999982        84631     54613.33
2910.207     0.999985        84631     65536.00
2910.207     0.999986        84631     72817.78
2910.207     0.999988        84631     81920.00
3018.751     0.999989        84632     93622.86
3018.751     1.000000        84632          inf

#[Mean = 29.576, StdDeviation = 202.840] #[Max = 3016.704, Total count = 84632] #[Buckets = 27, SubBuckets = 2048]

89756 requests in 2.99m, 469.72MB read Requests/sec: 501.12 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.660ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.640ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.534ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.659ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.12ms 8.12ms 503.81ms 99.68% Req/Sec 66.08 100.29 1.30k 72.18% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.78ms 75.000% 4.18ms 90.000% 4.62ms 99.000% 7.22ms 99.900% 53.02ms 99.990% 372.73ms 99.999% 504.06ms 100.000% 504.06ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.313     0.000000            1         1.00
   3.111     0.100000         4245         1.11
   3.323     0.200000         8494         1.25
   3.491     0.300000        12709         1.43
   3.639     0.400000        16947         1.67
   3.779     0.500000        21185         2.00
   3.849     0.550000        23306         2.22
   3.925     0.600000        25454         2.50
   4.001     0.650000        27594         2.86
   4.083     0.700000        29674         3.33
   4.183     0.750000        31834         4.00
   4.235     0.775000        32830         4.44
   4.299     0.800000        33949         5.00
   4.355     0.825000        34948         5.71
   4.427     0.850000        36055         6.67
   4.511     0.875000        37083         8.00
   4.559     0.887500        37608         8.89
   4.619     0.900000        38158        10.00
   4.679     0.912500        38662        11.43
   4.755     0.925000        39206        13.33
   4.839     0.937500        39718        16.00
   4.891     0.943750        39990        17.78
   4.951     0.950000        40250        20.00
   5.023     0.956250        40511        22.86
   5.119     0.962500        40780        26.67
   5.255     0.968750        41038        32.00
   5.347     0.971875        41170        35.56
   5.475     0.975000        41303        40.00
   5.667     0.978125        41434        45.71
   5.883     0.981250        41565        53.33
   6.231     0.984375        41699        64.00
   6.479     0.985938        41764        71.11
   6.731     0.987500        41829        80.00
   7.011     0.989062        41895        91.43
   7.419     0.990625        41964       106.67
   7.955     0.992188        42028       128.00
   8.263     0.992969        42063       142.22
   8.711     0.993750        42094       160.00
   9.495     0.994531        42127       182.86
  10.287     0.995313        42160       213.33
  11.047     0.996094        42193       256.00
  11.583     0.996484        42210       284.44
  12.431     0.996875        42226       320.00
  13.607     0.997266        42243       365.71
  19.231     0.997656        42259       426.67
  26.927     0.998047        42276       512.00
  31.759     0.998242        42284       568.89
  35.519     0.998437        42292       640.00
  38.943     0.998633        42301       731.43
  47.775     0.998828        42309       853.33
  53.631     0.999023        42317      1024.00
  56.639     0.999121        42321      1137.78
  58.751     0.999219        42325      1280.00
  68.159     0.999316        42330      1462.86
 112.383     0.999414        42334      1706.67
 269.567     0.999512        42338      2048.00
 271.359     0.999561        42340      2275.56
 298.751     0.999609        42343      2560.00
 311.295     0.999658        42344      2925.71
 326.655     0.999707        42346      3413.33
 329.983     0.999756        42348      4096.00
 339.711     0.999780        42349      4551.11
 343.039     0.999805        42350      5120.00
 343.807     0.999829        42351      5851.43
 362.495     0.999854        42352      6826.67
 372.735     0.999878        42354      8192.00
 372.735     0.999890        42354      9102.22
 372.735     0.999902        42354     10240.00
 430.591     0.999915        42355     11702.86
 430.591     0.999927        42355     13653.33
 453.631     0.999939        42356     16384.00
 453.631     0.999945        42356     18204.44
 453.631     0.999951        42356     20480.00
 487.679     0.999957        42357     23405.71
 487.679     0.999963        42357     27306.67
 487.679     0.999969        42357     32768.00
 487.679     0.999973        42357     36408.89
 487.679     0.999976        42357     40960.00
 504.063     0.999979        42358     46811.43
 504.063     1.000000        42358          inf

#[Mean = 4.119, StdDeviation = 8.122] #[Max = 503.808, Total count = 42358] #[Buckets = 27, SubBuckets = 2048]

44922 requests in 2.99m, 233.88MB read Requests/sec: 250.24 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 474.381ms, rate sampling interval: 3975ms Thread calibration: mean lat.: 397.511ms, rate sampling interval: 3508ms Thread calibration: mean lat.: 336.320ms, rate sampling interval: 3119ms Thread calibration: mean lat.: 290.679ms, rate sampling interval: 2725ms Thread Stats Avg Stdev Max +/- Stdev Latency 18.98ms 151.01ms 2.70s 98.60% Req/Sec 61.96 6.01 106.00 95.15% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.79ms 75.000% 4.21ms 90.000% 4.76ms 99.000% 413.18ms 99.900% 2.32s 99.990% 2.57s 99.999% 2.70s 100.000% 2.70s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.391     0.000000            1         1.00
   3.127     0.100000         4248         1.11
   3.331     0.200000         8496         1.25
   3.489     0.300000        12729         1.43
   3.639     0.400000        16943         1.67
   3.793     0.500000        21172         2.00
   3.871     0.550000        23300         2.22
   3.943     0.600000        25418         2.50
   4.025     0.650000        27560         2.86
   4.111     0.700000        29647         3.33
   4.211     0.750000        31764         4.00
   4.271     0.775000        32842         4.44
   4.335     0.800000        33893         5.00
   4.415     0.825000        34960         5.71
   4.507     0.850000        36017         6.67
   4.611     0.875000        37051         8.00
   4.675     0.887500        37594         8.89
   4.755     0.900000        38093        10.00
   4.855     0.912500        38622        11.43
   4.979     0.925000        39153        13.33
   5.187     0.937500        39680        16.00
   5.331     0.943750        39948        17.78
   5.579     0.950000        40209        20.00
   6.035     0.956250        40473        22.86
   6.971     0.962500        40738        26.67
   8.455     0.968750        41002        32.00
   9.231     0.971875        41134        35.56
  10.183     0.975000        41266        40.00
  11.239     0.978125        41401        45.71
  13.359     0.981250        41531        53.33
  94.015     0.984375        41663        64.00
 167.039     0.985938        41729        71.11
 248.575     0.987500        41795        80.00
 349.951     0.989062        41863        91.43
 458.495     0.990625        41929       106.67
 741.375     0.992188        41994       128.00
 875.519     0.992969        42027       142.22
1072.127     0.993750        42060       160.00
1218.559     0.994531        42093       182.86
1368.063     0.995313        42127       213.33
1568.767     0.996094        42159       256.00
1647.615     0.996484        42176       284.44
1732.607     0.996875        42192       320.00
1838.079     0.997266        42209       365.71
1901.567     0.997656        42225       426.67
2044.927     0.998047        42242       512.00
2095.103     0.998242        42250       568.89
2150.399     0.998437        42259       640.00
2213.887     0.998633        42269       731.43
2269.183     0.998828        42275       853.33
2322.431     0.999023        42283      1024.00
2342.911     0.999121        42287      1137.78
2371.583     0.999219        42291      1280.00
2418.687     0.999316        42296      1462.86
2437.119     0.999414        42300      1706.67
2471.935     0.999512        42304      2048.00
2502.655     0.999561        42306      2275.56
2512.895     0.999609        42308      2560.00
2523.135     0.999658        42310      2925.71
2531.327     0.999707        42313      3413.33
2533.375     0.999756        42316      4096.00
2533.375     0.999780        42316      4551.11
2533.375     0.999805        42316      5120.00
2537.471     0.999829        42317      5851.43
2545.663     0.999854        42318      6826.67
2570.239     0.999878        42319      8192.00
2574.335     0.999890        42320      9102.22
2574.335     0.999902        42320     10240.00
2588.671     0.999915        42321     11702.86
2588.671     0.999927        42321     13653.33
2613.247     0.999939        42322     16384.00
2613.247     0.999945        42322     18204.44
2613.247     0.999951        42322     20480.00
2660.351     0.999957        42323     23405.71
2660.351     0.999963        42323     27306.67
2660.351     0.999969        42323     32768.00
2660.351     0.999973        42323     36408.89
2660.351     0.999976        42323     40960.00
2697.215     0.999979        42324     46811.43
2697.215     1.000000        42324          inf

#[Mean = 18.976, StdDeviation = 151.010] #[Max = 2695.168, Total count = 42324] #[Buckets = 27, SubBuckets = 2048]

44888 requests in 2.99m, 233.69MB read Socket errors: connect 0, read 0, write 0, timeout 106 Requests/sec: 250.36 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 537.883ms, rate sampling interval: 4395ms Thread calibration: mean lat.: 479.803ms, rate sampling interval: 4016ms Thread calibration: mean lat.: 412.142ms, rate sampling interval: 3600ms Thread calibration: mean lat.: 348.973ms, rate sampling interval: 3162ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.30ms 25.95ms 884.74ms 99.21% Req/Sec 62.07 1.93 66.00 86.11% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.88ms 75.000% 4.41ms 90.000% 6.64ms 99.000% 15.09ms 99.900% 472.83ms 99.990% 681.98ms 99.999% 885.25ms 100.000% 885.25ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.313     0.000000            1         1.00
   3.157     0.100000         4236         1.11
   3.383     0.200000         8516         1.25
   3.563     0.300000        12714         1.43
   3.723     0.400000        16952         1.67
   3.881     0.500000        21196         2.00
   3.961     0.550000        23332         2.22
   4.053     0.600000        25452         2.50
   4.155     0.650000        27605         2.86
   4.267     0.700000        29682         3.33
   4.415     0.750000        31813         4.00
   4.507     0.775000        32869         4.44
   4.611     0.800000        33884         5.00
   4.759     0.825000        34965         5.71
   4.987     0.850000        36006         6.67
   5.463     0.875000        37062         8.00
   5.975     0.887500        37593         8.89
   6.643     0.900000        38120        10.00
   7.443     0.912500        38650        11.43
   8.263     0.925000        39181        13.33
   9.023     0.937500        39709        16.00
   9.407     0.943750        39972        17.78
   9.855     0.950000        40238        20.00
  10.279     0.956250        40501        22.86
  10.751     0.962500        40766        26.67
  11.263     0.968750        41030        32.00
  11.551     0.971875        41162        35.56
  11.863     0.975000        41299        40.00
  12.207     0.978125        41430        45.71
  12.567     0.981250        41559        53.33
  13.103     0.984375        41695        64.00
  13.415     0.985938        41759        71.11
  13.839     0.987500        41825        80.00
  14.359     0.989062        41890        91.43
  15.815     0.990625        41956       106.67
  34.559     0.992188        42023       128.00
  54.879     0.992969        42056       142.22
  91.455     0.993750        42089       160.00
 118.655     0.994531        42122       182.86
 162.687     0.995313        42156       213.33
 200.575     0.996094        42188       256.00
 220.927     0.996484        42205       284.44
 241.279     0.996875        42221       320.00
 271.359     0.997266        42238       365.71
 303.615     0.997656        42254       426.67
 360.703     0.998047        42271       512.00
 377.599     0.998242        42279       568.89
 405.759     0.998437        42287       640.00
 428.031     0.998633        42296       731.43
 452.607     0.998828        42304       853.33
 473.087     0.999023        42312      1024.00
 485.631     0.999121        42316      1137.78
 489.727     0.999219        42320      1280.00
 510.463     0.999316        42325      1462.86
 521.215     0.999414        42329      1706.67
 542.207     0.999512        42333      2048.00
 542.719     0.999561        42335      2275.56
 552.447     0.999609        42337      2560.00
 555.519     0.999658        42339      2925.71
 602.111     0.999707        42341      3413.33
 614.911     0.999756        42343      4096.00
 634.367     0.999780        42344      4551.11
 634.879     0.999805        42346      5120.00
 634.879     0.999829        42346      5851.43
 636.927     0.999854        42347      6826.67
 640.511     0.999878        42348      8192.00
 681.983     0.999890        42349      9102.22
 681.983     0.999902        42349     10240.00
 715.263     0.999915        42350     11702.86
 715.263     0.999927        42350     13653.33
 790.527     0.999939        42351     16384.00
 790.527     0.999945        42351     18204.44
 790.527     0.999951        42351     20480.00
 800.767     0.999957        42352     23405.71
 800.767     0.999963        42352     27306.67
 800.767     0.999969        42352     32768.00
 800.767     0.999973        42352     36408.89
 800.767     0.999976        42352     40960.00
 885.247     0.999979        42353     46811.43
 885.247     1.000000        42353          inf

#[Mean = 6.302, StdDeviation = 25.947] #[Max = 884.736, Total count = 42353] #[Buckets = 27, SubBuckets = 2048]

44917 requests in 2.99m, 233.86MB read Socket errors: connect 0, read 0, write 0, timeout 117 Requests/sec: 250.21 Transfer/sec: 1.30MB

1 server

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.925ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.609ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.631ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.642ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 3.88ms 0.98ms 26.40ms 92.22% Req/Sec 66.05 100.46 777.00 72.13% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.76ms 75.000% 4.14ms 90.000% 4.57ms 99.000% 7.37ms 99.900% 16.00ms 99.990% 24.17ms 99.999% 26.42ms 100.000% 26.42ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.491     0.000000            1         1.00
   3.125     0.100000         4262         1.11
   3.321     0.200000         8519         1.25
   3.481     0.300000        12732         1.43
   3.625     0.400000        16964         1.67
   3.761     0.500000        21185         2.00
   3.831     0.550000        23333         2.22
   3.901     0.600000        25436         2.50
   3.973     0.650000        27572         2.86
   4.047     0.700000        29657         3.33
   4.135     0.750000        31768         4.00
   4.187     0.775000        32848         4.44
   4.243     0.800000        33906         5.00
   4.311     0.825000        34975         5.71
   4.383     0.850000        36019         6.67
   4.471     0.875000        37108         8.00
   4.515     0.887500        37616         8.89
   4.567     0.900000        38138        10.00
   4.627     0.912500        38652        11.43
   4.699     0.925000        39189        13.33
   4.783     0.937500        39705        16.00
   4.839     0.943750        39976        17.78
   4.907     0.950000        40246        20.00
   4.987     0.956250        40503        22.86
   5.087     0.962500        40769        26.67
   5.255     0.968750        41030        32.00
   5.371     0.971875        41161        35.56
   5.523     0.975000        41296        40.00
   5.695     0.978125        41425        45.71
   5.951     0.981250        41557        53.33
   6.327     0.984375        41690        64.00
   6.615     0.985938        41757        71.11
   6.895     0.987500        41822        80.00
   7.167     0.989062        41888        91.43
   7.519     0.990625        41954       106.67
   7.987     0.992188        42023       128.00
   8.287     0.992969        42054       142.22
   8.599     0.993750        42087       160.00
   8.903     0.994531        42120       182.86
   9.359     0.995313        42153       213.33
   9.879     0.996094        42186       256.00
  10.383     0.996484        42204       284.44
  10.855     0.996875        42219       320.00
  11.311     0.997266        42236       365.71
  11.983     0.997656        42252       426.67
  13.655     0.998047        42269       512.00
  14.015     0.998242        42277       568.89
  14.687     0.998437        42285       640.00
  15.183     0.998633        42294       731.43
  15.703     0.998828        42302       853.33
  16.023     0.999023        42310      1024.00
  16.399     0.999121        42314      1137.78
  16.863     0.999219        42318      1280.00
  17.055     0.999316        42323      1462.86
  17.519     0.999414        42327      1706.67
  18.463     0.999512        42331      2048.00
  18.847     0.999561        42333      2275.56
  19.615     0.999609        42335      2560.00
  20.031     0.999658        42337      2925.71
  20.159     0.999707        42339      3413.33
  21.231     0.999756        42341      4096.00
  21.775     0.999780        42342      4551.11
  21.823     0.999805        42343      5120.00
  21.903     0.999829        42344      5851.43
  22.335     0.999854        42345      6826.67
  22.719     0.999878        42346      8192.00
  24.175     0.999890        42347      9102.22
  24.175     0.999902        42347     10240.00
  25.039     0.999915        42348     11702.86
  25.039     0.999927        42348     13653.33
  25.247     0.999939        42349     16384.00
  25.247     0.999945        42349     18204.44
  25.247     0.999951        42349     20480.00
  25.823     0.999957        42350     23405.71
  25.823     0.999963        42350     27306.67
  25.823     0.999969        42350     32768.00
  25.823     0.999973        42350     36408.89
  25.823     0.999976        42350     40960.00
  26.415     0.999979        42351     46811.43
  26.415     1.000000        42351          inf

#[Mean = 3.882, StdDeviation = 0.977] #[Max = 26.400, Total count = 42351] #[Buckets = 27, SubBuckets = 2048]

44915 requests in 2.99m, 233.84MB read Requests/sec: 250.52 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.623ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.609ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.597ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.618ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 2.88ms 669.90us 17.15ms 81.54% Req/Sec 65.71 99.27 555.00 72.00% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 2.81ms 75.000% 3.15ms 90.000% 3.51ms 99.000% 5.22ms 99.900% 9.32ms 99.990% 13.55ms 99.999% 17.17ms 100.000% 17.17ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.754     0.000000            1         1.00
   2.221     0.100000         4269         1.11
   2.405     0.200000         8526         1.25
   2.553     0.300000        12754         1.43
   2.687     0.400000        16935         1.67
   2.813     0.500000        21204         2.00
   2.877     0.550000        23335         2.22
   2.939     0.600000        25466         2.50
   3.003     0.650000        27560         2.86
   3.071     0.700000        29649         3.33
   3.149     0.750000        31754         4.00
   3.193     0.775000        32819         4.44
   3.241     0.800000        33913         5.00
   3.293     0.825000        34931         5.71
   3.353     0.850000        36016         6.67
   3.421     0.875000        37063         8.00
   3.461     0.887500        37582         8.89
   3.509     0.900000        38111        10.00
   3.557     0.912500        38635        11.43
   3.619     0.925000        39175        13.33
   3.693     0.937500        39692        16.00
   3.743     0.943750        39958        17.78
   3.791     0.950000        40235        20.00
   3.843     0.956250        40489        22.86
   3.913     0.962500        40750        26.67
   3.999     0.968750        41017        32.00
   4.057     0.971875        41148        35.56
   4.123     0.975000        41285        40.00
   4.207     0.978125        41413        45.71
   4.339     0.981250        41544        53.33
   4.563     0.984375        41676        64.00
   4.711     0.985938        41743        71.11
   4.887     0.987500        41809        80.00
   5.067     0.989062        41874        91.43
   5.343     0.990625        41942       106.67
   5.651     0.992188        42007       128.00
   5.807     0.992969        42041       142.22
   5.959     0.993750        42073       160.00
   6.207     0.994531        42106       182.86
   6.435     0.995313        42139       213.33
   6.839     0.996094        42172       256.00
   6.967     0.996484        42189       284.44
   7.147     0.996875        42205       320.00
   7.339     0.997266        42223       365.71
   7.543     0.997656        42239       426.67
   7.963     0.998047        42255       512.00
   8.079     0.998242        42263       568.89
   8.503     0.998437        42271       640.00
   8.783     0.998633        42280       731.43
   9.023     0.998828        42288       853.33
   9.351     0.999023        42296      1024.00
   9.447     0.999121        42300      1137.78
   9.911     0.999219        42304      1280.00
  10.047     0.999316        42309      1462.86
  10.159     0.999414        42313      1706.67
  10.311     0.999512        42318      2048.00
  10.335     0.999561        42319      2275.56
  10.567     0.999609        42321      2560.00
  11.503     0.999658        42323      2925.71
  12.175     0.999707        42325      3413.33
  12.647     0.999756        42327      4096.00
  12.743     0.999780        42328      4551.11
  12.751     0.999805        42329      5120.00
  12.791     0.999829        42330      5851.43
  12.991     0.999854        42331      6826.67
  13.327     0.999878        42332      8192.00
  13.551     0.999890        42333      9102.22
  13.551     0.999902        42333     10240.00
  13.575     0.999915        42334     11702.86
  13.575     0.999927        42334     13653.33
  14.903     0.999939        42335     16384.00
  14.903     0.999945        42335     18204.44
  14.903     0.999951        42335     20480.00
  16.719     0.999957        42336     23405.71
  16.719     0.999963        42336     27306.67
  16.719     0.999969        42336     32768.00
  16.719     0.999973        42336     36408.89
  16.719     0.999976        42336     40960.00
  17.167     0.999979        42337     46811.43
  17.167     1.000000        42337          inf

#[Mean = 2.881, StdDeviation = 0.670] #[Max = 17.152, Total count = 42337] #[Buckets = 27, SubBuckets = 2048]

44901 requests in 2.99m, 235.20MB read Requests/sec: 250.49 Transfer/sec: 1.31MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 257.054ms, rate sampling interval: 2349ms Thread calibration: mean lat.: 224.623ms, rate sampling interval: 2040ms Thread calibration: mean lat.: 369.132ms, rate sampling interval: 3317ms Thread calibration: mean lat.: 323.245ms, rate sampling interval: 2988ms Thread Stats Avg Stdev Max +/- Stdev Latency 7.76ms 10.76ms 366.08ms 97.06% Req/Sec 124.51 3.17 136.00 94.27% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 6.04ms 75.000% 8.81ms 90.000% 12.66ms 99.000% 31.18ms 99.900% 171.39ms 99.990% 343.30ms 99.999% 363.26ms 100.000% 366.33ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.549     0.000000            1         1.00
   3.597     0.100000         8472         1.11
   3.965     0.200000        16969         1.25
   4.343     0.300000        25454         1.43
   4.895     0.400000        33910         1.67
   6.039     0.500000        42360         2.00
   6.611     0.550000        46575         2.22
   7.111     0.600000        50823         2.50
   7.587     0.650000        55047         2.86
   8.131     0.700000        59296         3.33
   8.815     0.750000        63527         4.00
   9.215     0.775000        65666         4.44
   9.719     0.800000        67765         5.00
  10.247     0.825000        69864         5.71
  10.935     0.850000        71996         6.67
  11.647     0.875000        74100         8.00
  12.119     0.887500        75172         8.89
  12.663     0.900000        76219        10.00
  13.215     0.912500        77281        11.43
  13.847     0.925000        78336        13.33
  14.711     0.937500        79393        16.00
  15.231     0.943750        79918        17.78
  15.823     0.950000        80450        20.00
  16.559     0.956250        80985        22.86
  17.375     0.962500        81511        26.67
  18.255     0.968750        82039        32.00
  18.783     0.971875        82305        35.56
  19.583     0.975000        82564        40.00
  20.287     0.978125        82835        45.71
  21.375     0.981250        83097        53.33
  23.199     0.984375        83364        64.00
  24.767     0.985938        83491        71.11
  26.719     0.987500        83623        80.00
  29.327     0.989062        83755        91.43
  33.119     0.990625        83889       106.67
  35.903     0.992188        84020       128.00
  37.279     0.992969        84086       142.22
  41.151     0.993750        84152       160.00
  42.719     0.994531        84218       182.86
  47.135     0.995313        84286       213.33
  53.727     0.996094        84351       256.00
  57.855     0.996484        84386       284.44
  60.383     0.996875        84421       320.00
  64.767     0.997266        84450       365.71
  67.199     0.997656        84484       426.67
  96.063     0.998047        84516       512.00
 102.143     0.998242        84533       568.89
 109.119     0.998437        84549       640.00
 136.831     0.998633        84566       731.43
 171.135     0.998828        84583       853.33
 171.519     0.999023        84602      1024.00
 174.079     0.999121        84608      1137.78
 182.271     0.999219        84615      1280.00
 201.855     0.999316        84624      1462.86
 222.335     0.999414        84632      1706.67
 261.887     0.999512        84640      2048.00
 288.767     0.999561        84644      2275.56
 329.215     0.999609        84648      2560.00
 340.991     0.999658        84654      2925.71
 341.247     0.999707        84658      3413.33
 341.759     0.999756        84662      4096.00
 342.015     0.999780        84663      4551.11
 342.271     0.999805        84665      5120.00
 342.527     0.999829        84667      5851.43
 342.783     0.999854        84670      6826.67
 343.039     0.999878        84671      8192.00
 343.295     0.999890        84673      9102.22
 343.295     0.999902        84673     10240.00
 344.575     0.999915        84674     11702.86
 348.159     0.999927        84675     13653.33
 361.983     0.999939        84676     16384.00
 362.239     0.999945        84677     18204.44
 362.239     0.999951        84677     20480.00
 363.263     0.999957        84680     23405.71
 363.263     0.999963        84680     27306.67
 363.263     0.999969        84680     32768.00
 363.263     0.999973        84680     36408.89
 363.263     0.999976        84680     40960.00
 363.263     0.999979        84680     46811.43
 363.263     0.999982        84680     54613.33
 363.263     0.999985        84680     65536.00
 363.263     0.999986        84680     72817.78
 363.263     0.999988        84680     81920.00
 366.335     0.999989        84681     93622.86
 366.335     1.000000        84681          inf

#[Mean = 7.765, StdDeviation = 10.756] #[Max = 366.080, Total count = 84681] #[Buckets = 27, SubBuckets = 2048]

89575 requests in 2.99m, 468.72MB read Socket errors: connect 0, read 0, write 0, timeout 64 Requests/sec: 499.32 Transfer/sec: 2.61MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.077ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.662ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.627ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.657ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.29ms 3.02ms 73.79ms 87.72% Req/Sec 131.77 327.64 3.56k 98.85% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.23ms 75.000% 4.11ms 90.000% 8.37ms 99.000% 16.01ms 99.900% 26.27ms 99.990% 35.78ms 99.999% 37.38ms 100.000% 73.86ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.721     0.000000            1         1.00
   2.413     0.100000         8512         1.11
   2.659     0.200000        16978         1.25
   2.851     0.300000        25411         1.43
   3.031     0.400000        33858         1.67
   3.231     0.500000        42343         2.00
   3.339     0.550000        46560         2.22
   3.461     0.600000        50801         2.50
   3.607     0.650000        55047         2.86
   3.805     0.700000        59230         3.33
   4.111     0.750000        63468         4.00
   4.343     0.775000        65579         4.44
   4.691     0.800000        67689         5.00
   5.231     0.825000        69808         5.71
   6.075     0.850000        71917         6.67
   7.203     0.875000        74034         8.00
   7.767     0.887500        75093         8.89
   8.367     0.900000        76157        10.00
   9.023     0.912500        77212        11.43
   9.631     0.925000        78263        13.33
  10.247     0.937500        79319        16.00
  10.615     0.943750        79850        17.78
  10.983     0.950000        80379        20.00
  11.311     0.956250        80923        22.86
  11.775     0.962500        81437        26.67
  12.303     0.968750        81967        32.00
  12.743     0.971875        82231        35.56
  13.055     0.975000        82496        40.00
  13.423     0.978125        82756        45.71
  13.983     0.981250        83027        53.33
  14.471     0.984375        83287        64.00
  14.951     0.985938        83423        71.11
  15.231     0.987500        83550        80.00
  15.671     0.989062        83682        91.43
  16.367     0.990625        83813       106.67
  17.039     0.992188        83946       128.00
  17.343     0.992969        84013       142.22
  18.063     0.993750        84079       160.00
  19.023     0.994531        84144       182.86
  19.375     0.995313        84210       213.33
  20.063     0.996094        84281       256.00
  20.287     0.996484        84309       284.44
  20.815     0.996875        84342       320.00
  21.199     0.997266        84378       365.71
  21.695     0.997656        84408       426.67
  22.719     0.998047        84444       512.00
  22.799     0.998242        84458       568.89
  23.231     0.998437        84475       640.00
  23.455     0.998633        84494       731.43
  24.079     0.998828        84507       853.33
  26.367     0.999023        84525      1024.00
  26.415     0.999121        84533      1137.78
  26.431     0.999219        84540      1280.00
  26.495     0.999316        84549      1462.86
  26.559     0.999414        84558      1706.67
  28.559     0.999512        84565      2048.00
  30.143     0.999561        84569      2275.56
  31.119     0.999609        84573      2560.00
  35.551     0.999658        84579      2925.71
  35.647     0.999707        84585      3413.33
  35.679     0.999756        84588      4096.00
  35.679     0.999780        84588      4551.11
  35.711     0.999805        84592      5120.00
  35.711     0.999829        84592      5851.43
  35.775     0.999854        84598      6826.67
  35.775     0.999878        84598      8192.00
  35.775     0.999890        84598      9102.22
  35.775     0.999902        84598     10240.00
  35.807     0.999915        84600     11702.86
  35.807     0.999927        84600     13653.33
  35.839     0.999939        84602     16384.00
  35.839     0.999945        84602     18204.44
  35.839     0.999951        84602     20480.00
  36.415     0.999957        84603     23405.71
  36.415     0.999963        84603     27306.67
  37.247     0.999969        84604     32768.00
  37.247     0.999973        84604     36408.89
  37.247     0.999976        84604     40960.00
  37.375     0.999979        84605     46811.43
  37.375     0.999982        84605     54613.33
  37.375     0.999985        84605     65536.00
  37.375     0.999986        84605     72817.78
  37.375     0.999988        84605     81920.00
  73.855     0.999989        84606     93622.86
  73.855     1.000000        84606          inf

#[Mean = 4.288, StdDeviation = 3.018] #[Max = 73.792, Total count = 84606] #[Buckets = 27, SubBuckets = 2048]

89606 requests in 2.99m, 469.38MB read Requests/sec: 499.93 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.730ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.751ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.738ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.745ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 3.14ms 788.06us 21.73ms 80.52% Req/Sec 197.34 63.30 666.00 72.83% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.04ms 75.000% 3.47ms 90.000% 3.92ms 99.000% 5.59ms 99.900% 10.29ms 99.990% 18.35ms 99.999% 21.02ms 100.000% 21.74ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.656     0.000000            1         1.00
   2.361     0.100000        12685         1.11
   2.579     0.200000        25404         1.25
   2.749     0.300000        38090         1.43
   2.897     0.400000        50814         1.67
   3.039     0.500000        63471         2.00
   3.113     0.550000        69802         2.22
   3.191     0.600000        76140         2.50
   3.275     0.650000        82444         2.86
   3.369     0.700000        88831         3.33
   3.469     0.750000        95122         4.00
   3.525     0.775000        98273         4.44
   3.587     0.800000       101456         5.00
   3.655     0.825000       104657         5.71
   3.729     0.850000       107755         6.67
   3.817     0.875000       110968         8.00
   3.867     0.887500       112535         8.89
   3.925     0.900000       114120        10.00
   3.987     0.912500       115677        11.43
   4.065     0.925000       117264        13.33
   4.159     0.937500       118862        16.00
   4.215     0.943750       119659        17.78
   4.275     0.950000       120453        20.00
   4.347     0.956250       121223        22.86
   4.439     0.962500       122028        26.67
   4.543     0.968750       122807        32.00
   4.607     0.971875       123204        35.56
   4.679     0.975000       123592        40.00
   4.779     0.978125       123998        45.71
   4.899     0.981250       124389        53.33
   5.063     0.984375       124784        64.00
   5.171     0.985938       124977        71.11
   5.291     0.987500       125174        80.00
   5.483     0.989062       125374        91.43
   5.671     0.990625       125573       106.67
   5.907     0.992188       125767       128.00
   6.059     0.992969       125866       142.22
   6.247     0.993750       125967       160.00
   6.447     0.994531       126066       182.86
   6.711     0.995313       126165       213.33
   7.087     0.996094       126262       256.00
   7.255     0.996484       126312       284.44
   7.527     0.996875       126361       320.00
   7.783     0.997266       126411       365.71
   8.151     0.997656       126460       426.67
   8.535     0.998047       126510       512.00
   8.783     0.998242       126536       568.89
   8.975     0.998437       126559       640.00
   9.327     0.998633       126584       731.43
   9.799     0.998828       126609       853.33
  10.399     0.999023       126634      1024.00
  10.783     0.999121       126646      1137.78
  11.431     0.999219       126658      1280.00
  11.927     0.999316       126671      1462.86
  12.303     0.999414       126683      1706.67
  13.479     0.999512       126696      2048.00
  13.623     0.999561       126702      2275.56
  14.279     0.999609       126708      2560.00
  14.647     0.999658       126714      2925.71
  15.015     0.999707       126720      3413.33
  15.415     0.999756       126727      4096.00
  16.159     0.999780       126730      4551.11
  16.527     0.999805       126733      5120.00
  16.879     0.999829       126736      5851.43
  17.023     0.999854       126739      6826.67
  17.599     0.999878       126742      8192.00
  18.351     0.999890       126744      9102.22
  18.511     0.999902       126745     10240.00
  18.783     0.999915       126747     11702.86
  18.879     0.999927       126748     13653.33
  18.975     0.999939       126750     16384.00
  19.647     0.999945       126751     18204.44
  19.647     0.999951       126751     20480.00
  19.839     0.999957       126752     23405.71
  19.951     0.999963       126753     27306.67
  19.967     0.999969       126755     32768.00
  19.967     0.999973       126755     36408.89
  19.967     0.999976       126755     40960.00
  19.967     0.999979       126755     46811.43
  19.967     0.999982       126755     54613.33
  21.023     0.999985       126756     65536.00
  21.023     0.999986       126756     72817.78
  21.023     0.999988       126756     81920.00
  21.023     0.999989       126756     93622.86
  21.023     0.999991       126756    109226.67
  21.743     0.999992       126757    131072.00
  21.743     1.000000       126757          inf

#[Mean = 3.137, StdDeviation = 0.788] #[Max = 21.728, Total count = 126757] #[Buckets = 27, SubBuckets = 2048]

134385 requests in 2.98m, 703.68MB read Requests/sec: 751.48 Transfer/sec: 3.93MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.039ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.008ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.048ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.961ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.82ms 1.67ms 124.61ms 87.84% Req/Sec 197.61 68.79 800.00 69.21% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.44ms 75.000% 5.26ms 90.000% 6.53ms 99.000% 9.61ms 99.900% 21.02ms 99.990% 37.60ms 99.999% 84.67ms 100.000% 124.67ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.605     0.000000            1         1.00
   3.561     0.100000        12734         1.11
   3.825     0.200000        25448         1.25
   4.035     0.300000        38098         1.43
   4.235     0.400000        50910         1.67
   4.439     0.500000        63512         2.00
   4.555     0.550000        69907         2.22
   4.683     0.600000        76317         2.50
   4.835     0.650000        82568         2.86
   5.027     0.700000        88928         3.33
   5.263     0.750000        95215         4.00
   5.403     0.775000        98367         4.44
   5.579     0.800000       101545         5.00
   5.783     0.825000       104733         5.71
   6.007     0.850000       107880         6.67
   6.259     0.875000       111046         8.00
   6.395     0.887500       112664         8.89
   6.531     0.900000       114244        10.00
   6.671     0.912500       115832        11.43
   6.815     0.925000       117410        13.33
   6.979     0.937500       119017        16.00
   7.067     0.943750       119772        17.78
   7.171     0.950000       120567        20.00
   7.283     0.956250       121367        22.86
   7.423     0.962500       122156        26.67
   7.583     0.968750       122964        32.00
   7.683     0.971875       123348        35.56
   7.803     0.975000       123738        40.00
   7.935     0.978125       124134        45.71
   8.139     0.981250       124534        53.33
   8.431     0.984375       124929        64.00
   8.623     0.985938       125125        71.11
   8.879     0.987500       125323        80.00
   9.271     0.989062       125522        91.43
   9.855     0.990625       125722       106.67
  10.887     0.992188       125917       128.00
  11.551     0.992969       126016       142.22
  12.367     0.993750       126116       160.00
  13.143     0.994531       126215       182.86
  14.255     0.995313       126314       213.33
  15.335     0.996094       126414       256.00
  15.783     0.996484       126462       284.44
  16.447     0.996875       126512       320.00
  16.911     0.997266       126562       365.71
  17.471     0.997656       126613       426.67
  18.127     0.998047       126661       512.00
  18.591     0.998242       126685       568.89
  19.087     0.998437       126711       640.00
  19.663     0.998633       126736       731.43
  20.287     0.998828       126760       853.33
  21.039     0.999023       126785      1024.00
  21.343     0.999121       126798      1137.78
  21.551     0.999219       126809      1280.00
  22.047     0.999316       126822      1462.86
  22.639     0.999414       126834      1706.67
  23.647     0.999512       126847      2048.00
  24.015     0.999561       126853      2275.56
  25.007     0.999609       126859      2560.00
  25.615     0.999658       126865      2925.71
  27.167     0.999707       126871      3413.33
  28.511     0.999756       126878      4096.00
  29.679     0.999780       126881      4551.11
  31.583     0.999805       126884      5120.00
  32.255     0.999829       126887      5851.43
  33.023     0.999854       126890      6826.67
  35.519     0.999878       126893      8192.00
  37.599     0.999890       126896      9102.22
  37.599     0.999902       126896     10240.00
  40.895     0.999915       126898     11702.86
  41.471     0.999927       126899     13653.33
  48.767     0.999939       126901     16384.00
  50.463     0.999945       126902     18204.44
  50.463     0.999951       126902     20480.00
  55.039     0.999957       126903     23405.71
  56.991     0.999963       126904     27306.67
  64.383     0.999969       126905     32768.00
  64.383     0.999973       126905     36408.89
  64.383     0.999976       126905     40960.00
  78.783     0.999979       126906     46811.43
  78.783     0.999982       126906     54613.33
  84.671     0.999985       126907     65536.00
  84.671     0.999986       126907     72817.78
  84.671     0.999988       126907     81920.00
  84.671     0.999989       126907     93622.86
  84.671     0.999991       126907    109226.67
 124.671     0.999992       126908    131072.00
 124.671     1.000000       126908          inf

#[Mean = 4.817, StdDeviation = 1.665] #[Max = 124.608, Total count = 126908] #[Buckets = 27, SubBuckets = 2048]

134536 requests in 2.98m, 718.08MB read Requests/sec: 751.41 Transfer/sec: 4.01MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.919ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.032ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.003ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.996ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.64ms 32.23ms 1.05s 99.71% Req/Sec 264.01 119.71 3.56k 72.73% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.66ms 75.000% 5.64ms 90.000% 7.05ms 99.000% 14.79ms 99.900% 674.82ms 99.990% 994.82ms 99.999% 1.04s 100.000% 1.05s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.585     0.000000            1         1.00
   3.621     0.100000        16972         1.11
   3.923     0.200000        33894         1.25
   4.167     0.300000        50820         1.43
   4.407     0.400000        67693         1.67
   4.655     0.500000        84666         2.00
   4.795     0.550000        93117         2.22
   4.955     0.600000       101635         2.50
   5.135     0.650000       110052         2.86
   5.363     0.700000       118519         3.33
   5.635     0.750000       126903         4.00
   5.807     0.775000       131189         4.44
   6.003     0.800000       135345         5.00
   6.223     0.825000       139549         5.71
   6.467     0.850000       143826         6.67
   6.739     0.875000       148043         8.00
   6.891     0.887500       150152         8.89
   7.047     0.900000       152251        10.00
   7.223     0.912500       154356        11.43
   7.415     0.925000       156463        13.33
   7.651     0.937500       158591        16.00
   7.787     0.943750       159642        17.78
   7.943     0.950000       160700        20.00
   8.135     0.956250       161757        22.86
   8.375     0.962500       162811        26.67
   8.695     0.968750       163883        32.00
   8.895     0.971875       164397        35.56
   9.175     0.975000       164929        40.00
   9.511     0.978125       165455        45.71
  10.023     0.981250       165986        53.33
  11.063     0.984375       166508        64.00
  11.895     0.985938       166771        71.11
  12.831     0.987500       167035        80.00
  14.031     0.989062       167299        91.43
  15.455     0.990625       167566       106.67
  17.055     0.992188       167832       128.00
  17.855     0.992969       167961       142.22
  18.591     0.993750       168092       160.00
  19.615     0.994531       168225       182.86
  21.151     0.995313       168357       213.33
  23.039     0.996094       168492       256.00
  24.447     0.996484       168556       284.44
  26.703     0.996875       168621       320.00
  99.263     0.997266       168687       365.71
 240.383     0.997656       168753       426.67
 374.783     0.998047       168819       512.00
 429.055     0.998242       168852       568.89
 488.959     0.998437       168885       640.00
 553.471     0.998633       168918       731.43
 622.591     0.998828       168951       853.33
 680.447     0.999023       168984      1024.00
 724.991     0.999121       169001      1137.78
 755.711     0.999219       169017      1280.00
 795.135     0.999316       169034      1462.86
 823.295     0.999414       169050      1706.67
 858.111     0.999512       169067      2048.00
 870.911     0.999561       169075      2275.56
 884.223     0.999609       169083      2560.00
 902.143     0.999658       169092      2925.71
 920.063     0.999707       169100      3413.33
 934.911     0.999756       169108      4096.00
 940.543     0.999780       169112      4551.11
 949.759     0.999805       169116      5120.00
 961.023     0.999829       169121      5851.43
 970.239     0.999854       169125      6826.67
 985.087     0.999878       169129      8192.00
 992.767     0.999890       169131      9102.22
 997.375     0.999902       169133     10240.00
1002.495     0.999915       169135     11702.86
1007.615     0.999927       169137     13653.33
1012.735     0.999939       169139     16384.00
1015.295     0.999945       169140     18204.44
1018.367     0.999951       169141     20480.00
1019.903     0.999957       169142     23405.71
1022.975     0.999963       169143     27306.67
1027.583     0.999969       169144     32768.00
1033.215     0.999973       169145     36408.89
1033.215     0.999976       169145     40960.00
1038.335     0.999979       169146     46811.43
1038.335     0.999982       169146     54613.33
1043.455     0.999985       169147     65536.00
1043.455     0.999986       169147     72817.78
1043.455     0.999988       169147     81920.00
1047.551     0.999989       169148     93622.86
1047.551     0.999991       169148    109226.67
1047.551     0.999992       169148    131072.00
1047.551     0.999993       169148    145635.56
1047.551     0.999994       169148    163840.00
1054.719     0.999995       169149    187245.71
1054.719     1.000000       169149          inf

#[Mean = 6.635, StdDeviation = 32.233] #[Max = 1053.696, Total count = 169149] #[Buckets = 27, SubBuckets = 2048]

179302 requests in 2.98m, 0.94GB read Requests/sec: 1002.01 Transfer/sec: 5.39MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.666ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.674ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.704ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.665ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 3.48ms 1.12ms 21.70ms 79.89% Req/Sec 263.41 103.56 1.10k 75.59% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.26ms 75.000% 3.91ms 90.000% 4.74ms 99.000% 7.55ms 99.900% 11.66ms 99.990% 17.74ms 99.999% 20.74ms 100.000% 21.71ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.692     0.000000            1         1.00
   2.403     0.100000        17005         1.11
   2.665     0.200000        33987         1.25
   2.867     0.300000        50867         1.43
   3.055     0.400000        67787         1.67
   3.255     0.500000        84762         2.00
   3.361     0.550000        93248         2.22
   3.475     0.600000       101583         2.50
   3.605     0.650000       110115         2.86
   3.747     0.700000       118527         3.33
   3.909     0.750000       127046         4.00
   4.001     0.775000       131219         4.44
   4.107     0.800000       135500         5.00
   4.227     0.825000       139675         5.71
   4.363     0.850000       143964         6.67
   4.531     0.875000       148122         8.00
   4.631     0.887500       150279         8.89
   4.743     0.900000       152379        10.00
   4.867     0.912500       154477        11.43
   5.023     0.925000       156615        13.33
   5.219     0.937500       158716        16.00
   5.335     0.943750       159752        17.78
   5.471     0.950000       160820        20.00
   5.623     0.956250       161865        22.86
   5.803     0.962500       162930        26.67
   6.039     0.968750       163990        32.00
   6.175     0.971875       164518        35.56
   6.323     0.975000       165040        40.00
   6.487     0.978125       165570        45.71
   6.687     0.981250       166103        53.33
   6.915     0.984375       166628        64.00
   7.059     0.985938       166893        71.11
   7.243     0.987500       167154        80.00
   7.399     0.989062       167417        91.43
   7.635     0.990625       167683       106.67
   7.895     0.992188       167949       128.00
   8.051     0.992969       168080       142.22
   8.255     0.993750       168212       160.00
   8.455     0.994531       168344       182.86
   8.679     0.995313       168479       213.33
   8.943     0.996094       168608       256.00
   9.175     0.996484       168676       284.44
   9.375     0.996875       168740       320.00
   9.567     0.997266       168806       365.71
   9.863     0.997656       168877       426.67
  10.191     0.998047       168938       512.00
  10.463     0.998242       168973       568.89
  10.743     0.998437       169004       640.00
  11.015     0.998633       169038       731.43
  11.343     0.998828       169071       853.33
  11.719     0.999023       169103      1024.00
  11.911     0.999121       169120      1137.78
  12.183     0.999219       169136      1280.00
  12.519     0.999316       169153      1462.86
  12.895     0.999414       169169      1706.67
  13.535     0.999512       169186      2048.00
  13.735     0.999561       169194      2275.56
  13.943     0.999609       169202      2560.00
  14.255     0.999658       169211      2925.71
  14.511     0.999707       169219      3413.33
  15.183     0.999756       169227      4096.00
  15.615     0.999780       169231      4551.11
  15.983     0.999805       169235      5120.00
  16.591     0.999829       169240      5851.43
  16.975     0.999854       169245      6826.67
  17.311     0.999878       169248      8192.00
  17.551     0.999890       169250      9102.22
  17.743     0.999902       169252     10240.00
  17.903     0.999915       169254     11702.86
  18.111     0.999927       169256     13653.33
  18.687     0.999939       169258     16384.00
  18.703     0.999945       169259     18204.44
  18.831     0.999951       169260     20480.00
  19.375     0.999957       169262     23405.71
  19.375     0.999963       169262     27306.67
  19.439     0.999969       169263     32768.00
  19.919     0.999973       169264     36408.89
  19.919     0.999976       169264     40960.00
  20.335     0.999979       169265     46811.43
  20.335     0.999982       169265     54613.33
  20.735     0.999985       169266     65536.00
  20.735     0.999986       169266     72817.78
  20.735     0.999988       169266     81920.00
  21.407     0.999989       169267     93622.86
  21.407     0.999991       169267    109226.67
  21.407     0.999992       169267    131072.00
  21.407     0.999993       169267    145635.56
  21.407     0.999994       169267    163840.00
  21.711     0.999995       169268    187245.71
  21.711     1.000000       169268          inf

#[Mean = 3.482, StdDeviation = 1.123] #[Max = 21.696, Total count = 169268] #[Buckets = 27, SubBuckets = 2048]

179421 requests in 2.99m, 0.92GB read Requests/sec: 1001.79 Transfer/sec: 5.25MB

3 servers

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.121ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.941ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.829ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.836ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.60ms 3.68ms 75.58ms 96.34% Req/Sec 66.02 102.22 666.00 73.91% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.91ms 75.000% 4.36ms 90.000% 5.29ms 99.000% 30.11ms 99.900% 35.04ms 99.990% 59.42ms 99.999% 75.65ms 100.000% 75.65ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.465     0.000000            1         1.00
   3.237     0.100000         4247         1.11
   3.443     0.200000         8492         1.25
   3.605     0.300000        12708         1.43
   3.765     0.400000        16979         1.67
   3.909     0.500000        21189         2.00
   3.981     0.550000        23307         2.22
   4.057     0.600000        25445         2.50
   4.143     0.650000        27581         2.86
   4.239     0.700000        29643         3.33
   4.359     0.750000        31779         4.00
   4.431     0.775000        32852         4.44
   4.515     0.800000        33894         5.00
   4.619     0.825000        34954         5.71
   4.743     0.850000        36004         6.67
   4.927     0.875000        37046         8.00
   5.063     0.887500        37573         8.89
   5.287     0.900000        38100        10.00
   5.687     0.912500        38629        11.43
   6.343     0.925000        39158        13.33
   6.935     0.937500        39684        16.00
   7.263     0.943750        39952        17.78
   7.571     0.950000        40217        20.00
   7.891     0.956250        40482        22.86
   8.239     0.962500        40747        26.67
   8.687     0.968750        41008        32.00
   8.959     0.971875        41140        35.56
   9.287     0.975000        41271        40.00
   9.935     0.978125        41404        45.71
  11.663     0.981250        41538        53.33
  25.663     0.984375        41668        64.00
  28.399     0.985938        41735        71.11
  29.199     0.987500        41801        80.00
  29.823     0.989062        41867        91.43
  30.367     0.990625        41933       106.67
  30.911     0.992188        41999       128.00
  31.247     0.992969        42036       142.22
  31.615     0.993750        42066       160.00
  31.903     0.994531        42098       182.86
  32.207     0.995313        42131       213.33
  32.527     0.996094        42165       256.00
  32.735     0.996484        42181       284.44
  32.895     0.996875        42198       320.00
  33.151     0.997266        42217       365.71
  33.375     0.997656        42231       426.67
  33.727     0.998047        42247       512.00
  33.919     0.998242        42255       568.89
  34.111     0.998437        42264       640.00
  34.367     0.998633        42273       731.43
  34.623     0.998828        42280       853.33
  35.071     0.999023        42288      1024.00
  35.359     0.999121        42292      1137.78
  35.551     0.999219        42296      1280.00
  36.639     0.999316        42301      1462.86
  37.535     0.999414        42305      1706.67
  38.367     0.999512        42310      2048.00
  40.095     0.999561        42311      2275.56
  41.343     0.999609        42313      2560.00
  42.495     0.999658        42315      2925.71
  44.671     0.999707        42317      3413.33
  45.439     0.999756        42319      4096.00
  49.375     0.999780        42320      4551.11
  50.751     0.999805        42321      5120.00
  52.287     0.999829        42322      5851.43
  53.215     0.999854        42323      6826.67
  55.423     0.999878        42324      8192.00
  59.423     0.999890        42325      9102.22
  59.423     0.999902        42325     10240.00
  60.095     0.999915        42326     11702.86
  60.095     0.999927        42326     13653.33
  73.279     0.999939        42327     16384.00
  73.279     0.999945        42327     18204.44
  73.279     0.999951        42327     20480.00
  74.623     0.999957        42328     23405.71
  74.623     0.999963        42328     27306.67
  74.623     0.999969        42328     32768.00
  74.623     0.999973        42328     36408.89
  74.623     0.999976        42328     40960.00
  75.647     0.999979        42329     46811.43
  75.647     1.000000        42329          inf

#[Mean = 4.602, StdDeviation = 3.683] #[Max = 75.584, Total count = 42329] #[Buckets = 27, SubBuckets = 2048]

44893 requests in 2.99m, 233.72MB read Requests/sec: 250.45 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 9.505ms, rate sampling interval: 24ms Thread calibration: mean lat.: 6.753ms, rate sampling interval: 21ms Thread calibration: mean lat.: 5.401ms, rate sampling interval: 20ms Thread calibration: mean lat.: 7.067ms, rate sampling interval: 21ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.92ms 3.62ms 112.51ms 66.68% Req/Sec 63.98 165.91 1.60k 93.04% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 6.97ms 75.000% 9.12ms 90.000% 11.11ms 99.000% 15.60ms 99.900% 21.06ms 99.990% 99.14ms 99.999% 112.57ms 100.000% 112.57ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.751     0.000000            1         1.00
   2.719     0.100000         4235         1.11
   3.289     0.200000         8479         1.25
   4.347     0.300000        12707         1.43
   5.971     0.400000        16946         1.67
   6.971     0.500000        21188         2.00
   7.403     0.550000        23293         2.22
   7.827     0.600000        25420         2.50
   8.231     0.650000        27532         2.86
   8.655     0.700000        29646         3.33
   9.119     0.750000        31764         4.00
   9.367     0.775000        32842         4.44
   9.623     0.800000        33882         5.00
   9.927     0.825000        34950         5.71
  10.255     0.850000        35996         6.67
  10.663     0.875000        37069         8.00
  10.887     0.887500        37594         8.89
  11.111     0.900000        38116        10.00
  11.367     0.912500        38644        11.43
  11.647     0.925000        39176        13.33
  11.983     0.937500        39711        16.00
  12.175     0.943750        39961        17.78
  12.423     0.950000        40225        20.00
  12.695     0.956250        40493        22.86
  12.991     0.962500        40759        26.67
  13.343     0.968750        41019        32.00
  13.559     0.971875        41155        35.56
  13.799     0.975000        41284        40.00
  14.103     0.978125        41416        45.71
  14.487     0.981250        41550        53.33
  14.775     0.984375        41685        64.00
  14.919     0.985938        41747        71.11
  15.119     0.987500        41814        80.00
  15.415     0.989062        41880        91.43
  15.735     0.990625        41946       106.67
  16.087     0.992188        42016       128.00
  16.207     0.992969        42045       142.22
  16.383     0.993750        42078       160.00
  16.639     0.994531        42111       182.86
  16.863     0.995313        42145       213.33
  17.087     0.996094        42177       256.00
  17.279     0.996484        42197       284.44
  17.679     0.996875        42210       320.00
  17.983     0.997266        42227       365.71
  18.527     0.997656        42243       426.67
  19.007     0.998047        42260       512.00
  19.359     0.998242        42270       568.89
  19.951     0.998437        42276       640.00
  20.511     0.998633        42285       731.43
  20.911     0.998828        42293       853.33
  21.055     0.999023        42301      1024.00
  21.167     0.999121        42305      1137.78
  22.031     0.999219        42309      1280.00
  22.783     0.999316        42314      1462.86
  24.271     0.999414        42318      1706.67
  26.495     0.999512        42322      2048.00
  27.391     0.999561        42324      2275.56
  33.119     0.999609        42326      2560.00
  38.271     0.999658        42328      2925.71
  48.383     0.999707        42330      3413.33
  63.295     0.999756        42332      4096.00
  69.247     0.999780        42333      4551.11
  91.583     0.999805        42334      5120.00
  92.031     0.999829        42335      5851.43
  93.247     0.999854        42336      6826.67
  97.023     0.999878        42337      8192.00
  99.135     0.999890        42338      9102.22
  99.135     0.999902        42338     10240.00
 100.991     0.999915        42339     11702.86
 100.991     0.999927        42339     13653.33
 102.207     0.999939        42340     16384.00
 102.207     0.999945        42340     18204.44
 102.207     0.999951        42340     20480.00
 109.119     0.999957        42341     23405.71
 109.119     0.999963        42341     27306.67
 109.119     0.999969        42341     32768.00
 109.119     0.999973        42341     36408.89
 109.119     0.999976        42341     40960.00
 112.575     0.999979        42342     46811.43
 112.575     1.000000        42342          inf

#[Mean = 6.915, StdDeviation = 3.617] #[Max = 112.512, Total count = 42342] #[Buckets = 27, SubBuckets = 2048]

44846 requests in 2.99m, 234.91MB read Requests/sec: 250.17 Transfer/sec: 1.31MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.897ms, rate sampling interval: 16ms Thread calibration: mean lat.: 3.425ms, rate sampling interval: 13ms Thread calibration: mean lat.: 3.186ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.904ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 5.27ms 5.83ms 564.22ms 94.67% Req/Sec 130.66 111.25 0.89k 50.31% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.54ms 75.000% 7.51ms 90.000% 9.77ms 99.000% 14.15ms 99.900% 20.96ms 99.990% 295.93ms 99.999% 522.75ms 100.000% 564.73ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.697     0.000000            1         1.00
   2.457     0.100000         8490         1.11
   2.733     0.200000        16962         1.25
   2.965     0.300000        25403         1.43
   3.211     0.400000        33825         1.67
   3.541     0.500000        42307         2.00
   3.817     0.550000        46515         2.22
   4.395     0.600000        50745         2.50
   5.903     0.650000        54971         2.86
   6.859     0.700000        59194         3.33
   7.507     0.750000        63417         4.00
   7.835     0.775000        65550         4.44
   8.151     0.800000        67648         5.00
   8.503     0.825000        69770         5.71
   8.871     0.850000        71894         6.67
   9.279     0.875000        74009         8.00
   9.511     0.887500        75060         8.89
   9.775     0.900000        76121        10.00
  10.047     0.912500        77175        11.43
  10.375     0.925000        78214        13.33
  10.767     0.937500        79283        16.00
  10.991     0.943750        79806        17.78
  11.231     0.950000        80328        20.00
  11.495     0.956250        80858        22.86
  11.791     0.962500        81387        26.67
  12.103     0.968750        81913        32.00
  12.303     0.971875        82182        35.56
  12.495     0.975000        82443        40.00
  12.735     0.978125        82710        45.71
  12.999     0.981250        82975        53.33
  13.311     0.984375        83238        64.00
  13.487     0.985938        83372        71.11
  13.735     0.987500        83499        80.00
  13.999     0.989062        83632        91.43
  14.271     0.990625        83765       106.67
  14.607     0.992188        83894       128.00
  14.823     0.992969        83959       142.22
  15.111     0.993750        84025       160.00
  15.415     0.994531        84091       182.86
  15.775     0.995313        84157       213.33
  16.303     0.996094        84223       256.00
  16.527     0.996484        84256       284.44
  16.847     0.996875        84293       320.00
  17.215     0.997266        84324       365.71
  17.583     0.997656        84355       426.67
  18.175     0.998047        84388       512.00
  18.543     0.998242        84407       568.89
  18.959     0.998437        84421       640.00
  19.359     0.998633        84438       731.43
  20.207     0.998828        84454       853.33
  21.311     0.999023        84471      1024.00
  21.807     0.999121        84479      1137.78
  22.399     0.999219        84487      1280.00
  23.423     0.999316        84496      1462.86
  24.975     0.999414        84504      1706.67
  27.407     0.999512        84512      2048.00
  28.863     0.999561        84516      2275.56
  33.375     0.999609        84520      2560.00
  67.711     0.999658        84525      2925.71
  76.991     0.999707        84529      3413.33
 106.751     0.999756        84533      4096.00
 116.095     0.999780        84535      4551.11
 149.887     0.999805        84537      5120.00
 186.111     0.999829        84539      5851.43
 223.743     0.999854        84541      6826.67
 261.119     0.999878        84543      8192.00
 271.103     0.999890        84544      9102.22
 295.935     0.999902        84545     10240.00
 317.695     0.999915        84546     11702.86
 330.495     0.999927        84547     13653.33
 365.311     0.999939        84548     16384.00
 411.135     0.999945        84549     18204.44
 411.135     0.999951        84549     20480.00
 447.743     0.999957        84550     23405.71
 447.743     0.999963        84550     27306.67
 485.887     0.999969        84551     32768.00
 485.887     0.999973        84551     36408.89
 485.887     0.999976        84551     40960.00
 522.751     0.999979        84552     46811.43
 522.751     0.999982        84552     54613.33
 522.751     0.999985        84552     65536.00
 522.751     0.999986        84552     72817.78
 522.751     0.999988        84552     81920.00
 564.735     0.999989        84553     93622.86
 564.735     1.000000        84553          inf

#[Mean = 5.270, StdDeviation = 5.828] #[Max = 564.224, Total count = 84553] #[Buckets = 27, SubBuckets = 2048]

89677 requests in 2.98m, 469.75MB read Requests/sec: 500.96 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.886ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.853ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.887ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.816ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.78ms 6.98ms 236.16ms 93.49% Req/Sec 131.70 114.34 800.00 52.93% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.53ms 75.000% 6.90ms 90.000% 9.84ms 99.000% 35.23ms 99.900% 43.62ms 99.990% 195.33ms 99.999% 236.16ms 100.000% 236.29ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.553     0.000000            1         1.00
   3.465     0.100000         8462         1.11
   3.765     0.200000        16962         1.25
   3.999     0.300000        25421         1.43
   4.235     0.400000        33857         1.67
   4.531     0.500000        42350         2.00
   4.719     0.550000        46547         2.22
   4.983     0.600000        50790         2.50
   5.443     0.650000        55005         2.86
   6.171     0.700000        59227         3.33
   6.903     0.750000        63472         4.00
   7.223     0.775000        65562         4.44
   7.571     0.800000        67687         5.00
   7.967     0.825000        69808         5.71
   8.407     0.850000        71922         6.67
   8.951     0.875000        74028         8.00
   9.311     0.887500        75095         8.89
   9.839     0.900000        76139        10.00
  10.831     0.912500        77197        11.43
  12.223     0.925000        78256        13.33
  14.335     0.937500        79310        16.00
  16.639     0.943750        79837        17.78
  21.007     0.950000        80365        20.00
  28.479     0.956250        80896        22.86
  29.759     0.962500        81422        26.67
  30.863     0.968750        81951        32.00
  31.391     0.971875        82217        35.56
  31.919     0.975000        82485        40.00
  32.287     0.978125        82750        45.71
  32.863     0.981250        83016        53.33
  33.503     0.984375        83281        64.00
  33.919     0.985938        83407        71.11
  34.399     0.987500        83542        80.00
  34.879     0.989062        83674        91.43
  35.455     0.990625        83801       106.67
  36.223     0.992188        83934       128.00
  36.607     0.992969        84001       142.22
  36.991     0.993750        84067       160.00
  37.439     0.994531        84135       182.86
  38.015     0.995313        84201       213.33
  38.463     0.996094        84264       256.00
  38.751     0.996484        84297       284.44
  39.103     0.996875        84330       320.00
  39.551     0.997266        84363       365.71
  40.063     0.997656        84396       426.67
  40.895     0.998047        84430       512.00
  41.375     0.998242        84446       568.89
  42.015     0.998437        84463       640.00
  42.463     0.998633        84479       731.43
  43.007     0.998828        84495       853.33
  43.839     0.999023        84514      1024.00
  44.383     0.999121        84520      1137.78
  45.023     0.999219        84529      1280.00
  45.311     0.999316        84537      1462.86
  46.463     0.999414        84545      1706.67
  49.247     0.999512        84553      2048.00
  49.951     0.999561        84557      2275.56
  52.031     0.999609        84561      2560.00
  56.543     0.999658        84566      2925.71
  70.207     0.999707        84570      3413.33
  80.127     0.999756        84574      4096.00
 100.479     0.999780        84576      4551.11
 107.391     0.999805        84578      5120.00
 175.871     0.999829        84580      5851.43
 177.279     0.999854        84582      6826.67
 185.471     0.999878        84584      8192.00
 186.367     0.999890        84585      9102.22
 195.327     0.999902        84586     10240.00
 203.391     0.999915        84587     11702.86
 206.975     0.999927        84588     13653.33
 210.175     0.999939        84589     16384.00
 214.143     0.999945        84590     18204.44
 214.143     0.999951        84590     20480.00
 215.039     0.999957        84591     23405.71
 215.039     0.999963        84591     27306.67
 234.239     0.999969        84592     32768.00
 234.239     0.999973        84592     36408.89
 234.239     0.999976        84592     40960.00
 236.159     0.999979        84593     46811.43
 236.159     0.999982        84593     54613.33
 236.159     0.999985        84593     65536.00
 236.159     0.999986        84593     72817.78
 236.159     0.999988        84593     81920.00
 236.287     0.999989        84594     93622.86
 236.287     1.000000        84594          inf

#[Mean = 6.782, StdDeviation = 6.975] #[Max = 236.160, Total count = 84594] #[Buckets = 27, SubBuckets = 2048]

89718 requests in 2.98m, 469.51MB read Requests/sec: 501.09 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.802ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.842ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.815ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.819ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.78ms 10.37ms 555.01ms 96.87% Req/Sec 197.53 82.35 2.22k 59.16% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.61ms 75.000% 7.03ms 90.000% 11.23ms 99.000% 33.92ms 99.900% 145.15ms 99.990% 391.42ms 99.999% 462.59ms 100.000% 555.52ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.559     0.000000            1         1.00
   3.531     0.100000        12727         1.11
   3.821     0.200000        25459         1.25
   4.061     0.300000        38146         1.43
   4.307     0.400000        50921         1.67
   4.615     0.500000        63543         2.00
   4.823     0.550000        69844         2.22
   5.123     0.600000        76193         2.50
   5.595     0.650000        82536         2.86
   6.255     0.700000        88867         3.33
   7.027     0.750000        95210         4.00
   7.519     0.775000        98378         4.44
   8.087     0.800000       101535         5.00
   8.727     0.825000       104707         5.71
   9.423     0.850000       107903         6.67
  10.223     0.875000       111050         8.00
  10.703     0.887500       112647         8.89
  11.231     0.900000       114226        10.00
  11.767     0.912500       115817        11.43
  12.311     0.925000       117396        13.33
  12.943     0.937500       118990        16.00
  13.359     0.943750       119776        17.78
  13.887     0.950000       120572        20.00
  14.559     0.956250       121365        22.86
  15.575     0.962500       122160        26.67
  17.167     0.968750       122951        32.00
  18.463     0.971875       123344        35.56
  25.551     0.975000       123741        40.00
  29.391     0.978125       124141        45.71
  30.831     0.981250       124537        53.33
  32.063     0.984375       124930        64.00
  32.559     0.985938       125136        71.11
  33.055     0.987500       125334        80.00
  33.535     0.989062       125526        91.43
  34.175     0.990625       125724       106.67
  34.975     0.992188       125927       128.00
  35.423     0.992969       126023       142.22
  35.999     0.993750       126123       160.00
  36.543     0.994531       126219       182.86
  37.119     0.995313       126320       213.33
  37.855     0.996094       126420       256.00
  38.303     0.996484       126469       284.44
  38.783     0.996875       126518       320.00
  39.423     0.997266       126566       365.71
  40.703     0.997656       126616       426.67
  43.423     0.998047       126666       512.00
  46.335     0.998242       126690       568.89
  53.887     0.998437       126715       640.00
  69.439     0.998633       126740       731.43
  92.031     0.998828       126765       853.33
 150.527     0.999023       126790      1024.00
 169.215     0.999121       126802      1137.78
 187.775     0.999219       126814      1280.00
 207.359     0.999316       126827      1462.86
 237.695     0.999414       126839      1706.67
 261.887     0.999512       126852      2048.00
 295.679     0.999561       126858      2275.56
 299.775     0.999609       126864      2560.00
 309.503     0.999658       126870      2925.71
 316.415     0.999707       126876      3413.33
 332.799     0.999756       126883      4096.00
 339.967     0.999780       126886      4551.11
 343.551     0.999805       126889      5120.00
 347.391     0.999829       126892      5851.43
 368.383     0.999854       126895      6826.67
 384.767     0.999878       126898      8192.00
 391.423     0.999890       126900      9102.22
 400.639     0.999902       126902     10240.00
 401.663     0.999915       126903     11702.86
 402.687     0.999927       126904     13653.33
 410.111     0.999939       126906     16384.00
 417.535     0.999945       126907     18204.44
 417.535     0.999951       126907     20480.00
 421.631     0.999957       126908     23405.71
 442.111     0.999963       126909     27306.67
 448.511     0.999969       126910     32768.00
 448.511     0.999973       126910     36408.89
 448.511     0.999976       126910     40960.00
 451.071     0.999979       126911     46811.43
 451.071     0.999982       126911     54613.33
 462.591     0.999985       126912     65536.00
 462.591     0.999986       126912     72817.78
 462.591     0.999988       126912     81920.00
 462.591     0.999989       126912     93622.86
 462.591     0.999991       126912    109226.67
 555.519     0.999992       126913    131072.00
 555.519     1.000000       126913          inf

#[Mean = 6.781, StdDeviation = 10.370] #[Max = 555.008, Total count = 126913] #[Buckets = 27, SubBuckets = 2048]

134541 requests in 2.98m, 718.11MB read Requests/sec: 751.35 Transfer/sec: 4.01MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 93.400ms, rate sampling interval: 913ms Thread calibration: mean lat.: 2.765ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.751ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.726ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.10ms 2.22ms 80.83ms 84.80% Req/Sec 197.35 74.76 666.00 62.35% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.30ms 75.000% 4.42ms 90.000% 7.29ms 99.000% 11.41ms 99.900% 15.31ms 99.990% 61.15ms 99.999% 80.06ms 100.000% 80.89ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.681     0.000000            1         1.00
   2.443     0.100000        12729         1.11
   2.699     0.200000        25493         1.25
   2.899     0.300000        38209         1.43
   3.089     0.400000        50817         1.67
   3.301     0.500000        63486         2.00
   3.421     0.550000        69853         2.22
   3.559     0.600000        76196         2.50
   3.729     0.650000        82600         2.86
   3.977     0.700000        88899         3.33
   4.419     0.750000        95232         4.00
   4.835     0.775000        98419         4.44
   5.343     0.800000       101572         5.00
   5.839     0.825000       104755         5.71
   6.331     0.850000       107948         6.67
   6.795     0.875000       111104         8.00
   7.043     0.887500       112707         8.89
   7.287     0.900000       114280        10.00
   7.555     0.912500       115859        11.43
   7.863     0.925000       117451        13.33
   8.207     0.937500       119047        16.00
   8.391     0.943750       119858        17.78
   8.583     0.950000       120621        20.00
   8.815     0.956250       121409        22.86
   9.071     0.962500       122212        26.67
   9.383     0.968750       122998        32.00
   9.575     0.971875       123396        35.56
   9.775     0.975000       123791        40.00
  10.023     0.978125       124190        45.71
  10.311     0.981250       124584        53.33
  10.631     0.984375       124979        64.00
  10.823     0.985938       125180        71.11
  11.023     0.987500       125374        80.00
  11.255     0.989062       125572        91.43
  11.511     0.990625       125769       106.67
  11.799     0.992188       125968       128.00
  11.943     0.992969       126068       142.22
  12.095     0.993750       126169       160.00
  12.279     0.994531       126265       182.86
  12.527     0.995313       126365       213.33
  12.799     0.996094       126465       256.00
  12.967     0.996484       126516       284.44
  13.135     0.996875       126563       320.00
  13.359     0.997266       126612       365.71
  13.663     0.997656       126662       426.67
  13.999     0.998047       126713       512.00
  14.215     0.998242       126736       568.89
  14.399     0.998437       126762       640.00
  14.615     0.998633       126786       731.43
  14.975     0.998828       126811       853.33
  15.367     0.999023       126836      1024.00
  15.583     0.999121       126848      1137.78
  15.871     0.999219       126860      1280.00
  16.119     0.999316       126873      1462.86
  16.479     0.999414       126885      1706.67
  16.879     0.999512       126898      2048.00
  17.119     0.999561       126904      2275.56
  17.327     0.999609       126910      2560.00
  17.791     0.999658       126916      2925.71
  19.023     0.999707       126922      3413.33
  22.207     0.999756       126929      4096.00
  22.751     0.999780       126932      4551.11
  27.519     0.999805       126935      5120.00
  33.087     0.999829       126938      5851.43
  42.303     0.999854       126941      6826.67
  54.207     0.999878       126944      8192.00
  61.151     0.999890       126946      9102.22
  61.727     0.999902       126947     10240.00
  63.615     0.999915       126949     11702.86
  68.031     0.999927       126950     13653.33
  69.567     0.999939       126952     16384.00
  70.207     0.999945       126953     18204.44
  70.207     0.999951       126953     20480.00
  71.999     0.999957       126954     23405.71
  75.135     0.999963       126955     27306.67
  75.967     0.999969       126956     32768.00
  75.967     0.999973       126956     36408.89
  75.967     0.999976       126956     40960.00
  79.999     0.999979       126957     46811.43
  79.999     0.999982       126957     54613.33
  80.063     0.999985       126958     65536.00
  80.063     0.999986       126958     72817.78
  80.063     0.999988       126958     81920.00
  80.063     0.999989       126958     93622.86
  80.063     0.999991       126958    109226.67
  80.895     0.999992       126959    131072.00
  80.895     1.000000       126959          inf

#[Mean = 4.097, StdDeviation = 2.225] #[Max = 80.832, Total count = 126959] #[Buckets = 27, SubBuckets = 2048]

134587 requests in 2.99m, 704.74MB read Requests/sec: 751.46 Transfer/sec: 3.93MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.745ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.708ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.687ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.681ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.32ms 7.19ms 365.82ms 98.53% Req/Sec 263.21 111.26 3.33k 72.65% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.31ms 75.000% 4.42ms 90.000% 7.38ms 99.000% 12.36ms 99.900% 101.69ms 99.990% 297.73ms 99.999% 355.84ms 100.000% 366.08ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.674     0.000000            1         1.00
   2.393     0.100000        17016         1.11
   2.657     0.200000        33951         1.25
   2.875     0.300000        50889         1.43
   3.083     0.400000        67831         1.67
   3.313     0.500000        84654         2.00
   3.449     0.550000        93075         2.22
   3.607     0.600000       101518         2.50
   3.797     0.650000       110048         2.86
   4.043     0.700000       118441         3.33
   4.419     0.750000       126936         4.00
   4.687     0.775000       131133         4.44
   5.051     0.800000       135385         5.00
   5.507     0.825000       139595         5.71
   6.115     0.850000       143822         6.67
   6.771     0.875000       148046         8.00
   7.087     0.887500       150182         8.89
   7.383     0.900000       152288        10.00
   7.691     0.912500       154398        11.43
   8.027     0.925000       156523        13.33
   8.423     0.937500       158639        16.00
   8.663     0.943750       159699        17.78
   8.935     0.950000       160755        20.00
   9.239     0.956250       161797        22.86
   9.559     0.962500       162866        26.67
   9.895     0.968750       163915        32.00
  10.095     0.971875       164447        35.56
  10.311     0.975000       164968        40.00
  10.575     0.978125       165504        45.71
  10.911     0.981250       166023        53.33
  11.375     0.984375       166549        64.00
  11.639     0.985938       166821        71.11
  11.903     0.987500       167077        80.00
  12.191     0.989062       167343        91.43
  12.463     0.990625       167609       106.67
  12.759     0.992188       167878       128.00
  12.967     0.992969       168005       142.22
  13.215     0.993750       168136       160.00
  13.487     0.994531       168266       182.86
  13.927     0.995313       168400       213.33
  14.447     0.996094       168531       256.00
  14.719     0.996484       168599       284.44
  15.135     0.996875       168663       320.00
  15.759     0.997266       168732       365.71
  16.703     0.997656       168796       426.67
  18.415     0.998047       168861       512.00
  23.439     0.998242       168894       568.89
  40.223     0.998437       168927       640.00
  57.087     0.998633       168960       731.43
  81.151     0.998828       168993       853.33
 104.255     0.999023       169026      1024.00
 115.199     0.999121       169043      1137.78
 129.215     0.999219       169059      1280.00
 152.831     0.999316       169076      1462.86
 184.191     0.999414       169093      1706.67
 208.255     0.999512       169110      2048.00
 214.015     0.999561       169117      2275.56
 223.743     0.999609       169125      2560.00
 234.111     0.999658       169134      2925.71
 245.119     0.999707       169142      3413.33
 255.103     0.999756       169150      4096.00
 263.167     0.999780       169154      4551.11
 269.055     0.999805       169158      5120.00
 274.687     0.999829       169163      5851.43
 282.367     0.999854       169167      6826.67
 291.583     0.999878       169171      8192.00
 294.143     0.999890       169173      9102.22
 298.239     0.999902       169175     10240.00
 303.103     0.999915       169177     11702.86
 306.175     0.999927       169179     13653.33
 316.415     0.999939       169181     16384.00
 320.511     0.999945       169182     18204.44
 329.983     0.999951       169183     20480.00
 330.751     0.999957       169184     23405.71
 332.799     0.999963       169185     27306.67
 339.967     0.999969       169186     32768.00
 350.207     0.999973       169187     36408.89
 350.207     0.999976       169187     40960.00
 352.767     0.999979       169188     46811.43
 352.767     0.999982       169188     54613.33
 355.839     0.999985       169189     65536.00
 355.839     0.999986       169189     72817.78
 355.839     0.999988       169189     81920.00
 356.095     0.999989       169190     93622.86
 356.095     0.999991       169190    109226.67
 356.095     0.999992       169190    131072.00
 356.095     0.999993       169190    145635.56
 356.095     0.999994       169190    163840.00
 366.079     0.999995       169191    187245.71
 366.079     1.000000       169191          inf

#[Mean = 4.325, StdDeviation = 7.193] #[Max = 365.824, Total count = 169191] #[Buckets = 27, SubBuckets = 2048]

179346 requests in 2.98m, 0.92GB read Requests/sec: 1001.76 Transfer/sec: 5.25MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.887ms, rate sampling interval: 10ms Thread calibration: mean lat.: 4.088ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.867ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.973ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 8.73ms 45.92ms 983.04ms 99.09% Req/Sec 263.20 120.33 5.67k 73.55% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.66ms 75.000% 5.57ms 90.000% 6.91ms 99.000% 28.75ms 99.900% 801.28ms 99.990% 942.08ms 99.999% 980.48ms 100.000% 983.55ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.587     0.000000            1         1.00
   3.657     0.100000        16961         1.11
   3.951     0.200000        33861         1.25
   4.183     0.300000        50843         1.43
   4.411     0.400000        67830         1.67
   4.655     0.500000        84713         2.00
   4.795     0.550000        93255         2.22
   4.943     0.600000       101612         2.50
   5.115     0.650000       110031         2.86
   5.323     0.700000       118495         3.33
   5.575     0.750000       126954         4.00
   5.735     0.775000       131240         4.44
   5.911     0.800000       135413         5.00
   6.123     0.825000       139605         5.71
   6.359     0.850000       143864         6.67
   6.615     0.875000       148077         8.00
   6.759     0.887500       150216         8.89
   6.911     0.900000       152334        10.00
   7.075     0.912500       154429        11.43
   7.259     0.925000       156535        13.33
   7.483     0.937500       158655        16.00
   7.607     0.943750       159706        17.78
   7.763     0.950000       160770        20.00
   7.975     0.956250       161819        22.86
   8.271     0.962500       162877        26.67
   8.679     0.968750       163931        32.00
   8.951     0.971875       164468        35.56
   9.303     0.975000       164992        40.00
   9.775     0.978125       165522        45.71
  10.543     0.981250       166045        53.33
  12.239     0.984375       166573        64.00
  14.063     0.985938       166836        71.11
  16.543     0.987500       167102        80.00
  21.759     0.989062       167365        91.43
  44.831     0.990625       167629       106.67
 109.503     0.992188       167894       128.00
 163.071     0.992969       168026       142.22
 218.879     0.993750       168158       160.00
 286.207     0.994531       168290       182.86
 352.255     0.995313       168422       213.33
 426.751     0.996094       168556       256.00
 470.527     0.996484       168621       284.44
 513.535     0.996875       168688       320.00
 565.759     0.997266       168753       365.71
 615.423     0.997656       168819       426.67
 663.551     0.998047       168886       512.00
 692.223     0.998242       168918       568.89
 724.991     0.998437       168951       640.00
 747.007     0.998633       168985       731.43
 770.047     0.998828       169017       853.33
 802.815     0.999023       169050      1024.00
 816.639     0.999121       169067      1137.78
 832.511     0.999219       169083      1280.00
 843.775     0.999316       169100      1462.86
 857.599     0.999414       169116      1706.67
 873.471     0.999512       169134      2048.00
 875.007     0.999561       169141      2275.56
 882.687     0.999609       169149      2560.00
 889.855     0.999658       169158      2925.71
 897.535     0.999707       169166      3413.33
 909.311     0.999756       169174      4096.00
 911.359     0.999780       169178      4551.11
 914.431     0.999805       169183      5120.00
 919.551     0.999829       169187      5851.43
 921.087     0.999854       169191      6826.67
 934.911     0.999878       169195      8192.00
 939.519     0.999890       169197      9102.22
 944.639     0.999902       169199     10240.00
 946.175     0.999915       169201     11702.86
 952.831     0.999927       169203     13653.33
 955.903     0.999939       169205     16384.00
 958.463     0.999945       169207     18204.44
 958.463     0.999951       169207     20480.00
 963.583     0.999957       169208     23405.71
 965.119     0.999963       169209     27306.67
 970.751     0.999969       169210     32768.00
 974.847     0.999973       169211     36408.89
 974.847     0.999976       169211     40960.00
 979.455     0.999979       169212     46811.43
 979.455     0.999982       169212     54613.33
 980.479     0.999985       169213     65536.00
 980.479     0.999986       169213     72817.78
 980.479     0.999988       169213     81920.00
 983.039     0.999989       169214     93622.86
 983.039     0.999991       169214    109226.67
 983.039     0.999992       169214    131072.00
 983.039     0.999993       169214    145635.56
 983.039     0.999994       169214    163840.00
 983.551     0.999995       169215    187245.71
 983.551     1.000000       169215          inf

#[Mean = 8.732, StdDeviation = 45.920] #[Max = 983.040, Total count = 169215] #[Buckets = 27, SubBuckets = 2048]

179369 requests in 2.98m, 0.94GB read Requests/sec: 1001.69 Transfer/sec: 5.39MB

3 servers

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 4.990ms, rate sampling interval: 11ms Thread calibration: mean lat.: 3.976ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.842ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.767ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.73ms 3.47ms 191.87ms 95.84% Req/Sec 65.85 215.97 3.20k 96.68% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.09ms 75.000% 4.88ms 90.000% 6.46ms 99.000% 12.22ms 99.900% 36.35ms 99.990% 135.55ms 99.999% 192.00ms 100.000% 192.00ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.663     0.000000            1         1.00
   3.323     0.100000         4262         1.11
   3.553     0.200000         8470         1.25
   3.743     0.300000        12709         1.43
   3.911     0.400000        16965         1.67
   4.085     0.500000        21191         2.00
   4.187     0.550000        23321         2.22
   4.299     0.600000        25403         2.50
   4.447     0.650000        27554         2.86
   4.627     0.700000        29664         3.33
   4.879     0.750000        31764         4.00
   5.039     0.775000        32829         4.44
   5.227     0.800000        33891         5.00
   5.431     0.825000        34931         5.71
   5.683     0.850000        35994         6.67
   6.019     0.875000        37051         8.00
   6.215     0.887500        37583         8.89
   6.463     0.900000        38110        10.00
   6.723     0.912500        38639        11.43
   6.995     0.925000        39168        13.33
   7.427     0.937500        39699        16.00
   7.599     0.943750        39958        17.78
   7.851     0.950000        40223        20.00
   8.111     0.956250        40488        22.86
   8.455     0.962500        40753        26.67
   8.735     0.968750        41018        32.00
   8.943     0.971875        41150        35.56
   9.127     0.975000        41285        40.00
   9.407     0.978125        41414        45.71
   9.807     0.981250        41548        53.33
  10.215     0.984375        41677        64.00
  10.559     0.985938        41744        71.11
  11.199     0.987500        41809        80.00
  11.927     0.989062        41875        91.43
  12.527     0.990625        41942       106.67
  13.511     0.992188        42008       128.00
  14.831     0.992969        42041       142.22
  17.983     0.993750        42074       160.00
  20.911     0.994531        42107       182.86
  28.127     0.995313        42140       213.33
  29.999     0.996094        42173       256.00
  30.703     0.996484        42190       284.44
  31.055     0.996875        42206       320.00
  31.855     0.997266        42223       365.71
  32.271     0.997656        42239       426.67
  32.671     0.998047        42256       512.00
  33.023     0.998242        42264       568.89
  33.535     0.998437        42272       640.00
  34.335     0.998633        42282       731.43
  34.879     0.998828        42289       853.33
  36.831     0.999023        42297      1024.00
  38.655     0.999121        42301      1137.78
  39.679     0.999219        42305      1280.00
  49.759     0.999316        42310      1462.86
  50.431     0.999414        42314      1706.67
  63.263     0.999512        42318      2048.00
  68.031     0.999561        42320      2275.56
  68.223     0.999609        42322      2560.00
  70.847     0.999658        42324      2925.71
  75.711     0.999707        42326      3413.33
  86.463     0.999756        42328      4096.00
  87.615     0.999780        42329      4551.11
 105.983     0.999805        42330      5120.00
 124.607     0.999829        42331      5851.43
 127.423     0.999854        42332      6826.67
 130.367     0.999878        42333      8192.00
 135.551     0.999890        42334      9102.22
 135.551     0.999902        42334     10240.00
 140.927     0.999915        42335     11702.86
 140.927     0.999927        42335     13653.33
 176.127     0.999939        42336     16384.00
 176.127     0.999945        42336     18204.44
 176.127     0.999951        42336     20480.00
 181.247     0.999957        42337     23405.71
 181.247     0.999963        42337     27306.67
 181.247     0.999969        42337     32768.00
 181.247     0.999973        42337     36408.89
 181.247     0.999976        42337     40960.00
 191.999     0.999979        42338     46811.43
 191.999     1.000000        42338          inf

#[Mean = 4.728, StdDeviation = 3.466] #[Max = 191.872, Total count = 42338] #[Buckets = 27, SubBuckets = 2048]

44837 requests in 2.99m, 233.38MB read Requests/sec: 249.79 Transfer/sec: 1.30MB

~/wrk2/wrk --latency -R 250 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 9.282ms, rate sampling interval: 23ms Thread calibration: mean lat.: 6.733ms, rate sampling interval: 21ms Thread calibration: mean lat.: 5.250ms, rate sampling interval: 20ms Thread calibration: mean lat.: 4.640ms, rate sampling interval: 19ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.41ms 4.84ms 343.04ms 92.87% Req/Sec 63.98 95.21 421.00 76.99% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 6.69ms 75.000% 8.81ms 90.000% 10.67ms 99.000% 14.39ms 99.900% 18.46ms 99.990% 228.10ms 99.999% 343.30ms 100.000% 343.30ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.778     0.000000            1         1.00
   2.511     0.100000         4245         1.11
   2.841     0.200000         8467         1.25
   3.167     0.300000        12691         1.43
   4.595     0.400000        16922         1.67
   6.691     0.500000        21156         2.00
   7.119     0.550000        23284         2.22
   7.575     0.600000        25394         2.50
   7.987     0.650000        27504         2.86
   8.375     0.700000        29638         3.33
   8.807     0.750000        31755         4.00
   9.055     0.775000        32801         4.44
   9.319     0.800000        33857         5.00
   9.599     0.825000        34928         5.71
   9.903     0.850000        35955         6.67
  10.239     0.875000        37020         8.00
  10.439     0.887500        37547         8.89
  10.671     0.900000        38071        10.00
  10.903     0.912500        38606        11.43
  11.167     0.925000        39134        13.33
  11.471     0.937500        39657        16.00
  11.631     0.943750        39924        17.78
  11.823     0.950000        40184        20.00
  12.063     0.956250        40451        22.86
  12.295     0.962500        40712        26.67
  12.567     0.968750        40979        32.00
  12.727     0.971875        41114        35.56
  12.911     0.975000        41241        40.00
  13.111     0.978125        41373        45.71
  13.383     0.981250        41505        53.33
  13.711     0.984375        41638        64.00
  13.847     0.985938        41704        71.11
  14.039     0.987500        41770        80.00
  14.255     0.989062        41836        91.43
  14.479     0.990625        41902       106.67
  14.759     0.992188        41970       128.00
  14.919     0.992969        42002       142.22
  15.095     0.993750        42035       160.00
  15.287     0.994531        42068       182.86
  15.535     0.995313        42103       213.33
  15.807     0.996094        42133       256.00
  16.039     0.996484        42150       284.44
  16.271     0.996875        42167       320.00
  16.607     0.997266        42183       365.71
  16.863     0.997656        42199       426.67
  17.151     0.998047        42216       512.00
  17.343     0.998242        42224       568.89
  17.503     0.998437        42232       640.00
  17.903     0.998633        42241       731.43
  18.143     0.998828        42250       853.33
  18.495     0.999023        42257      1024.00
  18.911     0.999121        42261      1137.78
  19.247     0.999219        42265      1280.00
  19.823     0.999316        42270      1462.86
  20.799     0.999414        42275      1706.67
  20.959     0.999512        42278      2048.00
  21.743     0.999561        42280      2275.56
  22.143     0.999609        42282      2560.00
  22.783     0.999658        42284      2925.71
  23.375     0.999707        42286      3413.33
  24.607     0.999756        42288      4096.00
 138.623     0.999780        42289      4551.11
 155.519     0.999805        42290      5120.00
 162.303     0.999829        42291      5851.43
 177.663     0.999854        42292      6826.67
 197.375     0.999878        42293      8192.00
 228.095     0.999890        42294      9102.22
 228.095     0.999902        42294     10240.00
 264.959     0.999915        42295     11702.86
 264.959     0.999927        42295     13653.33
 282.111     0.999939        42296     16384.00
 282.111     0.999945        42296     18204.44
 282.111     0.999951        42296     20480.00
 307.199     0.999957        42297     23405.71
 307.199     0.999963        42297     27306.67
 307.199     0.999969        42297     32768.00
 307.199     0.999973        42297     36408.89
 307.199     0.999976        42297     40960.00
 343.295     0.999979        42298     46811.43
 343.295     1.000000        42298          inf

#[Mean = 6.412, StdDeviation = 4.841] #[Max = 343.040, Total count = 42298] #[Buckets = 27, SubBuckets = 2048]

44862 requests in 2.99m, 234.99MB read Requests/sec: 250.33 Transfer/sec: 1.31MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.978ms, rate sampling interval: 10ms Thread calibration: mean lat.: 8.551ms, rate sampling interval: 12ms Thread calibration: mean lat.: 5.285ms, rate sampling interval: 11ms Thread calibration: mean lat.: 4.019ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 10.34ms 10.87ms 345.09ms 89.06% Req/Sec 132.31 389.78 3.20k 97.43% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 6.64ms 75.000% 11.49ms 90.000% 22.53ms 99.000% 46.30ms 99.900% 85.57ms 99.990% 266.75ms 99.999% 339.97ms 100.000% 345.34ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.525     0.000000            1         1.00
   3.823     0.100000         8491         1.11
   4.279     0.200000        16930         1.25
   4.811     0.300000        25418         1.43
   5.567     0.400000        33860         1.67
   6.639     0.500000        42321         2.00
   7.239     0.550000        46562         2.22
   7.895     0.600000        50793         2.50
   8.647     0.650000        55015         2.86
   9.719     0.700000        59257         3.33
  11.487     0.750000        63487         4.00
  12.567     0.775000        65597         4.44
  13.855     0.800000        67723         5.00
  15.511     0.825000        69830         5.71
  17.039     0.850000        71950         6.67
  19.375     0.875000        74061         8.00
  20.783     0.887500        75120         8.89
  22.527     0.900000        76180        10.00
  24.351     0.912500        77235        11.43
  27.183     0.925000        78292        13.33
  29.727     0.937500        79351        16.00
  30.767     0.943750        79877        17.78
  31.791     0.950000        80414        20.00
  32.863     0.956250        80957        22.86
  34.079     0.962500        81473        26.67
  35.647     0.968750        81995        32.00
  36.511     0.971875        82258        35.56
  37.567     0.975000        82533        40.00
  38.591     0.978125        82793        45.71
  40.127     0.981250        83059        53.33
  41.535     0.984375        83316        64.00
  42.431     0.985938        83447        71.11
  43.871     0.987500        83582        80.00
  45.343     0.989062        83713        91.43
  46.591     0.990625        83846       106.67
  49.471     0.992188        83976       128.00
  51.103     0.992969        84042       142.22
  52.511     0.993750        84110       160.00
  54.975     0.994531        84175       182.86
  58.527     0.995313        84241       213.33
  60.031     0.996094        84307       256.00
  63.103     0.996484        84340       284.44
  65.055     0.996875        84374       320.00
  66.687     0.997266        84406       365.71
  67.711     0.997656        84439       426.67
  74.175     0.998047        84473       512.00
  75.007     0.998242        84492       568.89
  75.199     0.998437        84508       640.00
  78.463     0.998633        84526       731.43
  81.855     0.998828        84539       853.33
  85.759     0.999023        84555      1024.00
  85.887     0.999121        84567      1137.78
  86.015     0.999219        84575      1280.00
  87.295     0.999316        84581      1462.86
  92.607     0.999414        84589      1706.67
 118.335     0.999512        84596      2048.00
 198.015     0.999561        84600      2275.56
 216.447     0.999609        84604      2560.00
 235.775     0.999658        84609      2925.71
 237.183     0.999707        84613      3413.33
 246.271     0.999756        84617      4096.00
 246.527     0.999780        84619      4551.11
 246.911     0.999805        84621      5120.00
 247.295     0.999829        84623      5851.43
 254.975     0.999854        84625      6826.67
 264.703     0.999878        84627      8192.00
 266.495     0.999890        84628      9102.22
 266.751     0.999902        84629     10240.00
 272.127     0.999915        84630     11702.86
 273.663     0.999927        84631     13653.33
 276.479     0.999939        84632     16384.00
 289.535     0.999945        84633     18204.44
 289.535     0.999951        84633     20480.00
 293.119     0.999957        84634     23405.71
 293.119     0.999963        84634     27306.67
 304.895     0.999969        84635     32768.00
 304.895     0.999973        84635     36408.89
 304.895     0.999976        84635     40960.00
 339.967     0.999979        84636     46811.43
 339.967     0.999982        84636     54613.33
 339.967     0.999985        84636     65536.00
 339.967     0.999986        84636     72817.78
 339.967     0.999988        84636     81920.00
 345.343     0.999989        84637     93622.86
 345.343     1.000000        84637          inf

#[Mean = 10.341, StdDeviation = 10.870] #[Max = 345.088, Total count = 84637] #[Buckets = 27, SubBuckets = 2048]

89533 requests in 2.99m, 468.43MB read Requests/sec: 499.33 Transfer/sec: 2.61MB

~/wrk2/wrk --latency -R 500 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.746ms, rate sampling interval: 15ms Thread calibration: mean lat.: 3.432ms, rate sampling interval: 13ms Thread calibration: mean lat.: 3.226ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.051ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 5.35ms 8.83ms 765.95ms 99.02% Req/Sec 130.73 111.59 777.00 51.24% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.63ms 75.000% 7.37ms 90.000% 9.65ms 99.000% 14.18ms 99.900% 25.41ms 99.990% 516.86ms 99.999% 676.35ms 100.000% 766.46ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.748     0.000000            1         1.00
   2.493     0.100000         8479         1.11
   2.781     0.200000        16986         1.25
   3.021     0.300000        25413         1.43
   3.277     0.400000        33881         1.67
   3.635     0.500000        42344         2.00
   3.915     0.550000        46554         2.22
   4.479     0.600000        50774         2.50
   5.723     0.650000        55007         2.86
   6.711     0.700000        59236         3.33
   7.367     0.750000        63462         4.00
   7.679     0.775000        65578         4.44
   8.023     0.800000        67696         5.00
   8.391     0.825000        69833         5.71
   8.775     0.850000        71930         6.67
   9.175     0.875000        74067         8.00
   9.391     0.887500        75114         8.89
   9.647     0.900000        76171        10.00
   9.927     0.912500        77239        11.43
  10.255     0.925000        78289        13.33
  10.631     0.937500        79337        16.00
  10.871     0.943750        79858        17.78
  11.119     0.950000        80392        20.00
  11.383     0.956250        80919        22.86
  11.687     0.962500        81444        26.67
  12.007     0.968750        81976        32.00
  12.191     0.971875        82240        35.56
  12.375     0.975000        82507        40.00
  12.615     0.978125        82767        45.71
  12.927     0.981250        83033        53.33
  13.263     0.984375        83299        64.00
  13.479     0.985938        83428        71.11
  13.767     0.987500        83563        80.00
  14.023     0.989062        83692        91.43
  14.287     0.990625        83825       106.67
  14.615     0.992188        83956       128.00
  14.855     0.992969        84025       142.22
  15.111     0.993750        84088       160.00
  15.383     0.994531        84154       182.86
  15.823     0.995313        84221       213.33
  16.479     0.996094        84286       256.00
  16.847     0.996484        84320       284.44
  17.183     0.996875        84352       320.00
  17.823     0.997266        84386       365.71
  18.703     0.997656        84418       426.67
  19.855     0.998047        84452       512.00
  20.495     0.998242        84468       568.89
  21.295     0.998437        84485       640.00
  22.351     0.998633        84501       731.43
  23.839     0.998828        84517       853.33
  25.487     0.999023        84534      1024.00
  26.879     0.999121        84542      1137.78
  30.575     0.999219        84550      1280.00
  38.079     0.999316        84559      1462.86
  47.199     0.999414        84567      1706.67
  72.511     0.999512        84575      2048.00
 111.679     0.999561        84579      2275.56
 147.199     0.999609        84583      2560.00
 165.503     0.999658        84588      2925.71
 271.615     0.999707        84592      3413.33
 314.623     0.999756        84596      4096.00
 337.663     0.999780        84598      4551.11
 357.375     0.999805        84600      5120.00
 374.783     0.999829        84602      5851.43
 416.767     0.999854        84604      6826.67
 445.439     0.999878        84606      8192.00
 478.719     0.999890        84607      9102.22
 516.863     0.999902        84608     10240.00
 521.215     0.999915        84609     11702.86
 557.055     0.999927        84610     13653.33
 587.263     0.999939        84611     16384.00
 608.255     0.999945        84612     18204.44
 608.255     0.999951        84612     20480.00
 609.791     0.999957        84613     23405.71
 609.791     0.999963        84613     27306.67
 669.695     0.999969        84614     32768.00
 669.695     0.999973        84614     36408.89
 669.695     0.999976        84614     40960.00
 676.351     0.999979        84615     46811.43
 676.351     0.999982        84615     54613.33
 676.351     0.999985        84615     65536.00
 676.351     0.999986        84615     72817.78
 676.351     0.999988        84615     81920.00
 766.463     0.999989        84616     93622.86
 766.463     1.000000        84616          inf

#[Mean = 5.350, StdDeviation = 8.829] #[Max = 765.952, Total count = 84616] #[Buckets = 27, SubBuckets = 2048]

89740 requests in 2.98m, 470.09MB read Requests/sec: 501.06 Transfer/sec: 2.62MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 7.982ms, rate sampling interval: 10ms Thread calibration: mean lat.: 7.813ms, rate sampling interval: 10ms Thread calibration: mean lat.: 6.024ms, rate sampling interval: 10ms Thread calibration: mean lat.: 6.276ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 7.94ms 20.39ms 850.94ms 97.24% Req/Sec 197.36 88.75 2.00k 65.29% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.86ms 75.000% 7.68ms 90.000% 11.86ms 99.000% 36.80ms 99.900% 384.51ms 99.990% 586.75ms 99.999% 841.22ms 100.000% 851.46ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.593     0.000000            1         1.00
   3.623     0.100000        12691         1.11
   3.939     0.200000        25455         1.25
   4.207     0.300000        38133         1.43
   4.491     0.400000        50825         1.67
   4.863     0.500000        63519         2.00
   5.143     0.550000        69838         2.22
   5.563     0.600000        76182         2.50
   6.231     0.650000        82520         2.86
   6.975     0.700000        88838         3.33
   7.679     0.750000        95193         4.00
   8.079     0.775000        98350         4.44
   8.551     0.800000       101578         5.00
   9.095     0.825000       104697         5.71
   9.815     0.850000       107893         6.67
  10.751     0.875000       111058         8.00
  11.319     0.887500       112637         8.89
  11.863     0.900000       114216        10.00
  12.391     0.912500       115818        11.43
  12.975     0.925000       117392        13.33
  13.687     0.937500       118978        16.00
  14.111     0.943750       119771        17.78
  14.703     0.950000       120561        20.00
  15.735     0.956250       121353        22.86
  17.231     0.962500       122146        26.67
  19.647     0.968750       122937        32.00
  27.887     0.971875       123335        35.56
  30.015     0.975000       123731        40.00
  31.743     0.978125       124130        45.71
  32.895     0.981250       124535        53.33
  33.791     0.984375       124923        64.00
  34.335     0.985938       125123        71.11
  35.007     0.987500       125318        80.00
  36.095     0.989062       125517        91.43
  37.247     0.990625       125718       106.67
  38.367     0.992188       125915       128.00
  39.071     0.992969       126010       142.22
  40.319     0.993750       126109       160.00
  42.943     0.994531       126209       182.86
  55.071     0.995313       126308       213.33
  98.815     0.996094       126407       256.00
 121.023     0.996484       126456       284.44
 148.991     0.996875       126506       320.00
 180.863     0.997266       126556       365.71
 212.223     0.997656       126605       426.67
 251.391     0.998047       126655       512.00
 269.055     0.998242       126679       568.89
 298.751     0.998437       126705       640.00
 330.751     0.998633       126729       731.43
 353.535     0.998828       126754       853.33
 386.559     0.999023       126779      1024.00
 397.055     0.999121       126791      1137.78
 411.647     0.999219       126804      1280.00
 430.591     0.999316       126816      1462.86
 445.183     0.999414       126828      1706.67
 486.655     0.999512       126841      2048.00
 496.639     0.999561       126848      2275.56
 506.879     0.999609       126853      2560.00
 516.095     0.999658       126859      2925.71
 525.823     0.999707       126865      3413.33
 536.063     0.999756       126872      4096.00
 537.599     0.999780       126875      4551.11
 542.719     0.999805       126878      5120.00
 549.375     0.999829       126881      5851.43
 557.567     0.999854       126884      6826.67
 572.927     0.999878       126887      8192.00
 586.751     0.999890       126889      9102.22
 589.823     0.999902       126890     10240.00
 626.175     0.999915       126892     11702.86
 633.855     0.999927       126893     13653.33
 661.503     0.999939       126895     16384.00
 681.471     0.999945       126896     18204.44
 681.471     0.999951       126896     20480.00
 687.103     0.999957       126897     23405.71
 696.319     0.999963       126898     27306.67
 715.775     0.999969       126899     32768.00
 715.775     0.999973       126899     36408.89
 715.775     0.999976       126899     40960.00
 731.647     0.999979       126900     46811.43
 731.647     0.999982       126900     54613.33
 841.215     0.999985       126901     65536.00
 841.215     0.999986       126901     72817.78
 841.215     0.999988       126901     81920.00
 841.215     0.999989       126901     93622.86
 841.215     0.999991       126901    109226.67
 851.455     0.999992       126902    131072.00
 851.455     1.000000       126902          inf

#[Mean = 7.940, StdDeviation = 20.390] #[Max = 850.944, Total count = 126902] #[Buckets = 27, SubBuckets = 2048]

134529 requests in 2.98m, 717.95MB read Requests/sec: 751.46 Transfer/sec: 4.01MB

~/wrk2/wrk --latency -R 750 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.872ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.890ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.765ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.765ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 4.11ms 2.07ms 28.67ms 83.27% Req/Sec 197.44 74.14 666.00 62.28% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.33ms 75.000% 4.43ms 90.000% 7.27ms 99.000% 11.38ms 99.900% 14.94ms 99.990% 20.35ms 99.999% 26.62ms 100.000% 28.69ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.704     0.000000            1         1.00
   2.465     0.100000        12758         1.11
   2.719     0.200000        25472         1.25
   2.921     0.300000        38152         1.43
   3.117     0.400000        50878         1.67
   3.331     0.500000        63440         2.00
   3.459     0.550000        69865         2.22
   3.603     0.600000        76163         2.50
   3.775     0.650000        82493         2.86
   4.015     0.700000        88833         3.33
   4.435     0.750000        95166         4.00
   4.815     0.775000        98357         4.44
   5.335     0.800000       101521         5.00
   5.855     0.825000       104686         5.71
   6.339     0.850000       107875         6.67
   6.799     0.875000       111036         8.00
   7.027     0.887500       112621         8.89
   7.267     0.900000       114197        10.00
   7.527     0.912500       115801        11.43
   7.831     0.925000       117382        13.33
   8.171     0.937500       118968        16.00
   8.375     0.943750       119748        17.78
   8.591     0.950000       120553        20.00
   8.807     0.956250       121344        22.86
   9.071     0.962500       122130        26.67
   9.375     0.968750       122922        32.00
   9.535     0.971875       123314        35.56
   9.743     0.975000       123722        40.00
   9.951     0.978125       124111        45.71
  10.215     0.981250       124508        53.33
  10.567     0.984375       124898        64.00
  10.759     0.985938       125096        71.11
  10.983     0.987500       125297        80.00
  11.215     0.989062       125499        91.43
  11.471     0.990625       125695       106.67
  11.735     0.992188       125891       128.00
  11.911     0.992969       125991       142.22
  12.063     0.993750       126089       160.00
  12.231     0.994531       126190       182.86
  12.439     0.995313       126291       213.33
  12.735     0.996094       126385       256.00
  12.887     0.996484       126434       284.44
  13.071     0.996875       126485       320.00
  13.263     0.997266       126536       365.71
  13.503     0.997656       126584       426.67
  13.855     0.998047       126634       512.00
  13.991     0.998242       126658       568.89
  14.191     0.998437       126683       640.00
  14.431     0.998633       126709       731.43
  14.687     0.998828       126732       853.33
  14.999     0.999023       126757      1024.00
  15.271     0.999121       126769      1137.78
  15.471     0.999219       126781      1280.00
  15.727     0.999316       126794      1462.86
  16.015     0.999414       126806      1706.67
  16.367     0.999512       126819      2048.00
  16.623     0.999561       126826      2275.56
  16.943     0.999609       126831      2560.00
  17.375     0.999658       126837      2925.71
  17.743     0.999707       126844      3413.33
  18.047     0.999756       126850      4096.00
  18.303     0.999780       126853      4551.11
  18.463     0.999805       126856      5120.00
  18.831     0.999829       126860      5851.43
  19.039     0.999854       126862      6826.67
  19.567     0.999878       126865      8192.00
  20.351     0.999890       126867      9102.22
  20.447     0.999902       126868     10240.00
  20.847     0.999915       126870     11702.86
  21.423     0.999927       126871     13653.33
  22.255     0.999939       126873     16384.00
  22.431     0.999945       126874     18204.44
  22.431     0.999951       126874     20480.00
  22.799     0.999957       126875     23405.71
  23.103     0.999963       126876     27306.67
  24.575     0.999969       126877     32768.00
  24.575     0.999973       126877     36408.89
  24.575     0.999976       126877     40960.00
  25.087     0.999979       126878     46811.43
  25.087     0.999982       126878     54613.33
  26.623     0.999985       126879     65536.00
  26.623     0.999986       126879     72817.78
  26.623     0.999988       126879     81920.00
  26.623     0.999989       126879     93622.86
  26.623     0.999991       126879    109226.67
  28.687     0.999992       126880    131072.00
  28.687     1.000000       126880          inf

#[Mean = 4.107, StdDeviation = 2.072] #[Max = 28.672, Total count = 126880] #[Buckets = 27, SubBuckets = 2048]

134508 requests in 2.98m, 704.32MB read Requests/sec: 751.63 Transfer/sec: 3.94MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUserBook_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 2.784ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.733ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.746ms, rate sampling interval: 10ms Thread calibration: mean lat.: 2.676ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 6.44ms 35.97ms 1.55s 99.26% Req/Sec 263.22 120.35 5.00k 71.76% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 3.37ms 75.000% 4.47ms 90.000% 7.52ms 99.000% 14.28ms 99.900% 601.09ms 99.990% 1.25s 99.999% 1.50s 100.000% 1.55s

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   1.639     0.000000            1         1.00
   2.435     0.100000        16919         1.11
   2.703     0.200000        33877         1.25
   2.923     0.300000        50872         1.43
   3.139     0.400000        67719         1.67
   3.369     0.500000        84575         2.00
   3.511     0.550000        93029         2.22
   3.669     0.600000       101574         2.50
   3.859     0.650000       109957         2.86
   4.107     0.700000       118420         3.33
   4.475     0.750000       126864         4.00
   4.755     0.775000       131080         4.44
   5.131     0.800000       135300         5.00
   5.651     0.825000       139535         5.71
   6.299     0.850000       143747         6.67
   6.931     0.875000       147989         8.00
   7.223     0.887500       150094         8.89
   7.523     0.900000       152222        10.00
   7.819     0.912500       154325        11.43
   8.155     0.925000       156448        13.33
   8.535     0.937500       158572        16.00
   8.751     0.943750       159630        17.78
   9.015     0.950000       160679        20.00
   9.303     0.956250       161736        22.86
   9.647     0.962500       162779        26.67
  10.087     0.968750       163836        32.00
  10.359     0.971875       164369        35.56
  10.679     0.975000       164887        40.00
  11.159     0.978125       165423        45.71
  11.727     0.981250       165943        53.33
  12.367     0.984375       166472        64.00
  12.719     0.985938       166738        71.11
  13.183     0.987500       167000        80.00
  13.743     0.989062       167263        91.43
  14.935     0.990625       167527       106.67
  27.343     0.992188       167791       128.00
  58.335     0.992969       167923       142.22
  90.367     0.993750       168056       160.00
 125.567     0.994531       168188       182.86
 173.951     0.995313       168321       213.33
 221.183     0.996094       168452       256.00
 251.775     0.996484       168518       284.44
 282.879     0.996875       168584       320.00
 312.063     0.997266       168651       365.71
 342.527     0.997656       168716       426.67
 374.527     0.998047       168782       512.00
 394.239     0.998242       168815       568.89
 413.183     0.998437       168848       640.00
 450.815     0.998633       168881       731.43
 527.871     0.998828       168914       853.33
 608.767     0.999023       168947      1024.00
 655.359     0.999121       168964      1137.78
 703.487     0.999219       168980      1280.00
 760.831     0.999316       168997      1462.86
 803.839     0.999414       169014      1706.67
 880.127     0.999512       169030      2048.00
 898.047     0.999561       169038      2275.56
 929.791     0.999609       169046      2560.00
 978.431     0.999658       169055      2925.71
1015.807     0.999707       169063      3413.33
1046.527     0.999756       169071      4096.00
1080.319     0.999780       169075      4551.11
1103.871     0.999805       169079      5120.00
1141.759     0.999829       169084      5851.43
1172.479     0.999854       169088      6826.67
1209.343     0.999878       169092      8192.00
1238.015     0.999890       169094      9102.22
1253.375     0.999902       169096     10240.00
1274.879     0.999915       169098     11702.86
1297.407     0.999927       169100     13653.33
1325.055     0.999939       169102     16384.00
1355.775     0.999945       169103     18204.44
1372.159     0.999951       169104     20480.00
1379.327     0.999957       169105     23405.71
1393.663     0.999963       169106     27306.67
1415.167     0.999969       169107     32768.00
1423.359     0.999973       169108     36408.89
1423.359     0.999976       169108     40960.00
1451.007     0.999979       169109     46811.43
1451.007     0.999982       169109     54613.33
1497.087     0.999985       169110     65536.00
1497.087     0.999986       169110     72817.78
1497.087     0.999988       169110     81920.00
1505.279     0.999989       169111     93622.86
1505.279     0.999991       169111    109226.67
1505.279     0.999992       169111    131072.00
1505.279     0.999993       169111    145635.56
1505.279     0.999994       169111    163840.00
1549.311     0.999995       169112    187245.71
1549.311     1.000000       169112          inf

#[Mean = 6.444, StdDeviation = 35.971] #[Max = 1548.288, Total count = 169112] #[Buckets = 27, SubBuckets = 2048]

179268 requests in 2.98m, 0.92GB read Requests/sec: 1001.98 Transfer/sec: 5.25MB

~/wrk2/wrk --latency -R 1000 -d 3m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 3m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 3.888ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.902ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.872ms, rate sampling interval: 10ms Thread calibration: mean lat.: 3.814ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 8.95ms 38.96ms 753.15ms 98.68% Req/Sec 262.59 112.91 2.60k 72.23% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 4.60ms 75.000% 5.62ms 90.000% 7.20ms 99.000% 133.63ms 99.900% 570.37ms 99.990% 696.32ms 99.999% 748.54ms 100.000% 753.66ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.575     0.000000            1         1.00
   3.617     0.100000        16945         1.11
   3.907     0.200000        33914         1.25
   4.135     0.300000        51009         1.43
   4.363     0.400000        67950         1.67
   4.603     0.500000        84819         2.00
   4.739     0.550000        93216         2.22
   4.899     0.600000       101642         2.50
   5.087     0.650000       110066         2.86
   5.323     0.700000       118540         3.33
   5.619     0.750000       127079         4.00
   5.807     0.775000       131310         4.44
   6.027     0.800000       135490         5.00
   6.271     0.825000       139757         5.71
   6.535     0.850000       143989         6.67
   6.835     0.875000       148169         8.00
   7.003     0.887500       150290         8.89
   7.195     0.900000       152388        10.00
   7.423     0.912500       154503        11.43
   7.711     0.925000       156636        13.33
   8.115     0.937500       158742        16.00
   8.383     0.943750       159819        17.78
   8.703     0.950000       160861        20.00
   9.103     0.956250       161914        22.86
   9.631     0.962500       162976        26.67
  10.335     0.968750       164026        32.00
  10.807     0.971875       164556        35.56
  11.423     0.975000       165086        40.00
  12.215     0.978125       165614        45.71
  13.631     0.981250       166143        53.33
  19.711     0.984375       166672        64.00
  32.095     0.985938       166936        71.11
  64.095     0.987500       167201        80.00
 106.431     0.989062       167466        91.43
 158.719     0.990625       167730       106.67
 224.255     0.992188       167995       128.00
 261.887     0.992969       168129       142.22
 297.215     0.993750       168259       160.00
 331.263     0.994531       168392       182.86
 367.359     0.995313       168524       213.33
 404.223     0.996094       168656       256.00
 421.887     0.996484       168722       284.44
 442.879     0.996875       168788       320.00
 465.407     0.997266       168856       365.71
 484.095     0.997656       168921       426.67
 504.319     0.998047       168988       512.00
 517.375     0.998242       169020       568.89
 530.943     0.998437       169053       640.00
 543.743     0.998633       169086       731.43
 555.007     0.998828       169119       853.33
 574.463     0.999023       169154      1024.00
 582.143     0.999121       169169      1137.78
 590.335     0.999219       169185      1280.00
 598.527     0.999316       169202      1462.86
 607.743     0.999414       169218      1706.67
 616.959     0.999512       169235      2048.00
 623.103     0.999561       169243      2275.56
 628.223     0.999609       169252      2560.00
 633.855     0.999658       169260      2925.71
 643.071     0.999707       169268      3413.33
 653.823     0.999756       169276      4096.00
 660.479     0.999780       169280      4551.11
 665.087     0.999805       169284      5120.00
 673.791     0.999829       169289      5851.43
 680.447     0.999854       169293      6826.67
 689.151     0.999878       169298      8192.00
 695.807     0.999890       169299      9102.22
 697.343     0.999902       169301     10240.00
 702.975     0.999915       169303     11702.86
 707.071     0.999927       169305     13653.33
 710.143     0.999939       169307     16384.00
 717.823     0.999945       169308     18204.44
 730.623     0.999951       169309     20480.00
 733.183     0.999957       169310     23405.71
 735.743     0.999963       169311     27306.67
 738.303     0.999969       169312     32768.00
 738.815     0.999973       169313     36408.89
 738.815     0.999976       169313     40960.00
 740.351     0.999979       169314     46811.43
 740.351     0.999982       169314     54613.33
 748.543     0.999985       169315     65536.00
 748.543     0.999986       169315     72817.78
 748.543     0.999988       169315     81920.00
 751.615     0.999989       169316     93622.86
 751.615     0.999991       169316    109226.67
 751.615     0.999992       169316    131072.00
 751.615     0.999993       169316    145635.56
 751.615     0.999994       169316    163840.00
 753.663     0.999995       169317    187245.71
 753.663     1.000000       169317          inf

#[Mean = 8.951, StdDeviation = 38.957] #[Max = 753.152, Total count = 169317] #[Buckets = 27, SubBuckets = 2048]

179470 requests in 2.99m, 0.94GB read Requests/sec: 1001.51 Transfer/sec: 5.39MB

10 minutes run!

~/wrk2/wrk --latency -R 250 -d 10m -c 128 -t 4 -s getAnnotationsForUser_Paged.lua http://172.31.36.34:5000 Running 10m test @ http://172.31.36.34:5000 4 threads and 128 connections Thread calibration: mean lat.: 13.049ms, rate sampling interval: 64ms Thread calibration: mean lat.: 11.004ms, rate sampling interval: 61ms Thread calibration: mean lat.: 6.842ms, rate sampling interval: 10ms Thread calibration: mean lat.: 7.888ms, rate sampling interval: 10ms Thread Stats Avg Stdev Max +/- Stdev Latency 9.86ms 13.13ms 503.81ms 87.90% Req/Sec 65.43 103.65 0.90k 77.34% Latency Distribution (HdrHistogram - Recorded Latency) 50.000% 6.16ms 75.000% 9.68ms 90.000% 28.13ms 99.000% 44.67ms 99.900% 170.37ms 99.990% 384.26ms 99.999% 486.91ms 100.000% 504.06ms

Detailed Percentile spectrum: Value Percentile TotalCount 1/(1-Percentile)

   2.273     0.000000            1         1.00
   3.309     0.100000        14777         1.11
   3.691     0.200000        29531         1.25
   4.107     0.300000        44255         1.43
   4.783     0.400000        58987         1.67
   6.155     0.500000        73719         2.00
   6.847     0.550000        81089         2.22
   7.435     0.600000        88469         2.50
   7.991     0.650000        95830         2.86
   8.703     0.700000       103201         3.33
   9.679     0.750000       110544         4.00
  10.367     0.775000       114238         4.44
  11.191     0.800000       117928         5.00
  12.199     0.825000       121591         5.71
  13.767     0.850000       125276         6.67
  18.159     0.875000       128957         8.00
  26.607     0.887500       130802         8.89
  28.127     0.900000       132653        10.00
  29.199     0.912500       134492        11.43
  30.159     0.925000       136325        13.33
  31.151     0.937500       138193        16.00
  31.615     0.943750       139108        17.78
  32.111     0.950000       140026        20.00
  32.655     0.956250       140929        22.86
  33.279     0.962500       141860        26.67
  33.983     0.968750       142798        32.00
  34.399     0.971875       143242        35.56
  34.911     0.975000       143689        40.00
  35.551     0.978125       144167        45.71
  36.415     0.981250       144621        53.33
  37.567     0.984375       145076        64.00
  38.463     0.985938       145306        71.11
  39.519     0.987500       145531        80.00
  41.663     0.989062       145763        91.43
  47.103     0.990625       145995       106.67
  50.335     0.992188       146222       128.00
  52.479     0.992969       146337       142.22
  53.983     0.993750       146452       160.00
  55.359     0.994531       146569       182.86
  56.607     0.995313       146684       213.33
  57.919     0.996094       146798       256.00
  58.879     0.996484       146855       284.44
  60.191     0.996875       146913       320.00
  64.479     0.997266       146971       365.71
  79.743     0.997656       147028       426.67
 101.439     0.998047       147086       512.00
 116.223     0.998242       147114       568.89
 129.791     0.998437       147143       640.00
 142.079     0.998633       147172       731.43
 161.151     0.998828       147201       853.33
 172.543     0.999023       147230      1024.00
 180.351     0.999121       147244      1137.78
 189.183     0.999219       147258      1280.00
 198.399     0.999316       147273      1462.86
 208.511     0.999414       147287      1706.67
 226.943     0.999512       147302      2048.00
 238.975     0.999561       147309      2275.56
 248.703     0.999609       147316      2560.00
 260.607     0.999658       147323      2925.71
 274.943     0.999707       147330      3413.33
 302.847     0.999756       147338      4096.00
 310.527     0.999780       147341      4551.11
 334.591     0.999805       147345      5120.00
 344.063     0.999829       147348      5851.43
 354.303     0.999854       147352      6826.67
 366.847     0.999878       147356      8192.00
 376.575     0.999890       147357      9102.22
 386.815     0.999902       147359     10240.00
 399.615     0.999915       147361     11702.86
 409.343     0.999927       147363     13653.33
 426.239     0.999939       147365     16384.00
 426.239     0.999945       147365     18204.44
 432.895     0.999951       147366     20480.00
 433.407     0.999957       147367     23405.71
 451.839     0.999963       147368     27306.67
 453.887     0.999969       147369     32768.00
 453.887     0.999973       147369     36408.89
 459.263     0.999976       147370     40960.00
 459.263     0.999979       147370     46811.43
 463.615     0.999982       147371     54613.33
 463.615     0.999985       147371     65536.00
 463.615     0.999986       147371     72817.78
 486.911     0.999988       147372     81920.00
 486.911     0.999989       147372     93622.86
 486.911     0.999991       147372    109226.67
 486.911     0.999992       147372    131072.00
 486.911     0.999993       147372    145635.56
 504.063     0.999994       147373    163840.00
 504.063     1.000000       147373          inf

#[Mean = 9.863, StdDeviation = 13.126] #[Max = 503.808, Total count = 147373] #[Buckets = 27, SubBuckets = 2048]

149937 requests in 9.99m, 803.34MB read Requests/sec: 250.08 Transfer/sec: 1.34MB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment