Skip to content

Instantly share code, notes, and snippets.

@mdcallag

mdcallag/gist:4005e27018655a818bece429b07cfaf4

Created September 14, 2018 18:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mdcallag/4005e27018655a818bece429b07cfaf4 to your computer and use it in GitHub Desktop.
Save mdcallag/4005e27018655a818bece429b07cfaf4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
cached test
server use discard (SSD TRIM)
measured from iostat
dyn=1; cpri=3; for j in 1 2 4 8 12 16 24 32 40 48 64 ; do echo j $j, pri $cpri, dyn $dyn; head -3 hw.j$j.pri$cpri.dyn$dyn | tail -1 ; done; done; done | grep -v pri | awk '{ printf "%.0f\t%.0f\t%.0f\t%.0f\n", $3, $5, $6, $7 }'
rMB/s wMB/s rGB wGB
0 196 0 1374
0 260 0 1376
0 624 0 1308
0 966 0 1402
0 1146 0 1453
0 1218 0 1465
0 1313 0 1489
0 1366 0 1507
0 1429 0 1528
0 1447 0 1526
0 1424 0 1491
measured from compaction IO stats
* wGB.1, rGB.2 - GB for writes, reads during fillrandom
* wGB.2, rGB.2 - GB for writes, reads while waiting for compaction to finish
* wGB.t - wGB.1 + wGB.2
* rGB.t - rGB.1 + rGB.2
dyn=1; cpri=3; for j in 1 2 4 8 12 16 24 32 40 48 64 ; do echo j $j, pri $cpri, dyn $dyn; tail -1 hw.j$j.pri$cpri.dyn$dyn | tail -1 ; done; done; done | grep -v pri
wGB.1 wGB.2 rGB.1 rGB.2 wGB.t rGB.t
455.07 48.37 413.39 54.03 :: 503.44 467.42
459.79 45.52 418.31 51.03 :: 505.31 469.34
440.86 38.11 400.77 41.94 :: 478.97 442.71
494.85 29.76 456.33 31.86 :: 524.61 488.19
523.73 23.00 485.74 24.58 :: 546.73 510.32
531.14 20.28 493.42 21.58 :: 551.42 515.00
547.68 16.23 510.24 17.24 :: 563.91 527.48
556.64 16.05 519.12 17.05 :: 572.69 536.17
562.75 18.99 525.14 20.17 :: 581.74 545.31
557.46 20.55 519.50 21.95 :: 578.01 541.45
549.00 13.40 511.09 14.21 :: 562.40 525.30
jobs=1
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 15/0 493.04 MB 3.8 0.0 0.0 0.0 55.4 55.4 0.0 1.0 0.0 140.6 403 1726 0.234 0 0
L4 38/1 2.11 GB 8.2 144.5 50.8 93.6 142.9 49.3 0.0 2.8 115.9 114.6 1277 70 18.241 651M 6974K
L5 256/5 14.66 GB 5.9 202.1 50.3 151.7 195.5 43.8 0.6 3.9 122.8 118.8 1685 768 2.195 911M 29M
L6 459/0 24.42 GB 0.0 66.8 30.0 36.8 61.2 24.4 0.0 2.0 125.4 115.0 546 470 1.161 305M 22M
Sum 768/6 41.68 GB 0.0 413.4 131.2 282.2 455.1 172.9 0.6 8.2 108.2 119.1 3911 3034 1.289 1869M 58M
Int 0/0 0.00 KB 0.0 0.5 0.1 0.4 0.9 0.5 0.0 2.2 73.9 127.1 8 16 0.472 2462K 111K
Uptime(secs): 3587.3 total, 6.8 interval
Flush(GB): cumulative 55.401, interval 0.417
Cumulative compaction: 455.07 GB write, 129.90 MB/s write, 413.39 GB read, 118.00 MB/s read, 3911.3 seconds
Stalls(count): 1591 level0_slowdown, 252 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 79 memtable_compaction, 0 memtable_slowdown, interval 2 total count
jobs=2
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 65.74 MB 0.5 0.0 0.0 0.0 55.4 55.4 0.0 1.0 0.0 140.0 405 1726 0.235 0 0
L4 39/0 2.21 GB 8.8 146.4 51.3 95.1 144.8 49.7 0.0 2.8 116.6 115.3 1286 71 18.111 660M 7039K
L5 248/0 14.65 GB 6.0 204.1 50.9 153.2 197.5 44.3 0.4 3.9 123.8 119.8 1689 772 2.187 921M 29M
L6 453/0 24.55 GB 0.0 67.8 30.3 37.5 62.1 24.6 0.0 2.1 126.8 116.1 547 477 1.147 309M 23M
Sum 742/0 41.47 GB 0.0 418.3 132.5 285.8 459.8 174.0 0.4 8.3 109.1 119.9 3927 3046 1.289 1891M 59M
Int 0/0 0.00 KB 0.0 2.2 0.8 1.5 2.2 0.8 0.0 69.7 115.5 115.8 20 2 9.896 10M 114K
Uptime(secs): 3598.5 total, 7.6 interval
Flush(GB): cumulative 55.402, interval 0.032
Cumulative compaction: 459.79 GB write, 130.84 MB/s write, 418.31 GB read, 119.04 MB/s read, 3926.9 seconds
Stalls(count): 1592 level0_slowdown, 253 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 75 memtable_compaction, 0 memtable_slowdown, interval 2 total count
jobs=4
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 18/0 591.66 MB 4.5 65.3 0.0 65.3 120.5 55.2 0.0 2.2 64.8 119.5 1032 1961 0.526 294M 1085K
L3 10/0 630.59 MB 2.5 17.2 9.0 8.3 17.1 8.8 0.0 1.9 114.3 113.0 154 10 15.447 77M 841K
L4 22/3 1.37 GB 4.1 69.1 44.7 24.3 67.7 43.3 5.1 1.5 115.7 113.3 611 50 12.229 311M 6231K
L5 145/9 8.49 GB 2.7 110.6 41.7 68.9 107.7 38.8 9.0 2.6 117.9 114.8 961 563 1.707 499M 13M
L6 532/0 29.05 GB 0.0 138.5 39.6 98.9 127.9 29.0 0.0 3.2 122.5 113.2 1157 633 1.828 635M 44M
Sum 727/12 40.09 GB 0.0 400.8 135.0 265.7 440.9 175.1 14.1 8.0 104.8 115.3 3917 3217 1.217 1818M 65M
Int 0/0 0.00 KB 0.0 1.1 0.6 0.6 1.4 0.8 0.0 5.3 99.1 119.4 12 13 0.895 5081K 116K
Uptime(secs): 1236.7 total, 3.9 interval
Flush(GB): cumulative 55.401, interval 0.257
Cumulative compaction: 440.86 GB write, 365.03 MB/s write, 400.77 GB read, 331.83 MB/s read, 3916.7 seconds
Stalls(count): 625 level0_slowdown, 518 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 199 memtable_compaction, 0 memtable_slowdown, interval 0 total count
jobs=8
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 9/4 1.28 GB 3.5 66.6 0.0 66.6 121.9 55.3 0.0 2.2 63.6 116.4 1072 2091 0.513 300M 621K
L3 18/0 1.11 GB 4.4 22.2 12.8 9.4 21.9 12.5 0.0 1.7 110.4 108.8 206 11 18.708 99M 1352K
L4 28/5 1.72 GB 4.5 61.1 37.5 23.6 60.1 36.5 11.4 1.6 113.8 112.0 550 27 20.367 275M 4308K
L5 62/4 3.21 GB 1.0 86.5 39.9 46.6 85.1 38.5 9.7 2.1 92.5 91.0 957 504 1.900 390M 6001K
L6 533/0 31.20 GB 0.0 219.9 45.3 174.7 205.9 31.2 0.0 4.5 119.6 111.9 1883 742 2.538 1011M 60M
Sum 650/13 38.52 GB 0.0 456.3 135.5 320.8 494.9 174.0 21.1 8.9 100.1 108.5 4669 3375 1.383 2077M 72M
Int 0/0 0.00 KB 0.0 2.9 1.4 1.5 3.0 1.5 1.7 15.6 99.4 104.3 30 8 3.689 12M 219K
Uptime(secs): 1022.0 total, 4.0 interval
Flush(GB): cumulative 55.369, interval 0.193
Cumulative compaction: 494.85 GB write, 495.82 MB/s write, 456.33 GB read, 457.22 MB/s read, 4668.9 seconds
Stalls(count): 251 level0_slowdown, 251 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 153 memtable_compaction, 0 memtable_slowdown, interval 1 total count
jobs=12
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 14/13 1.19 GB 0.2 66.7 0.0 66.7 122.0 55.3 0.0 2.2 63.0 115.3 1083 2110 0.513 300M 685K
L3 16/16 994.43 MB 0.0 24.4 15.1 9.3 24.1 14.8 0.0 1.6 108.3 107.0 230 14 16.443 109M 1279K
L4 9/4 580.21 MB 1.0 53.3 35.0 18.4 52.6 34.2 13.3 1.5 113.0 111.5 483 32 15.107 240M 2997K
L5 68/13 3.66 GB 0.9 100.1 42.4 57.7 98.6 40.9 9.0 2.3 80.6 79.4 1271 570 2.229 451M 6458K
L6 534/0 31.61 GB 0.0 241.3 46.4 194.9 226.5 31.6 0.0 4.9 113.3 106.3 2182 756 2.886 1110M 63M
Sum 641/46 38.00 GB 0.0 485.7 138.8 346.9 523.7 176.8 22.3 9.5 94.8 102.2 5249 3482 1.508 2212M 74M
Int 0/0 0.00 KB 0.0 3.7 1.4 2.3 4.1 1.9 0.9 8.6 78.5 87.6 48 27 1.792 16M 224K
Uptime(secs): 983.5 total, 8.7 interval
Flush(GB): cumulative 55.401, interval 0.481
Cumulative compaction: 523.73 GB write, 545.31 MB/s write, 485.74 GB read, 505.75 MB/s read, 5249.4 seconds
Stalls(count): 124 level0_slowdown, 124 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 197 memtable_compaction, 0 memtable_slowdown, interval 2 total count
jobs=16
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 15/6 2.30 GB 3.6 69.5 0.0 69.5 124.8 55.3 0.0 2.3 63.7 114.3 1118 2130 0.525 313M 780K
L3 3/3 193.40 MB 0.0 22.4 14.5 7.9 22.1 14.2 0.0 1.5 108.8 107.5 211 13 16.200 100M 1119K
L4 4/0 257.87 MB 0.8 52.2 34.0 18.2 51.5 33.3 14.1 1.5 112.9 111.4 473 28 16.895 235M 2907K
L5 61/0 3.14 GB 1.0 100.0 42.4 57.6 98.5 41.0 8.9 2.3 75.3 74.2 1359 559 2.432 450M 6294K
L6 543/0 31.84 GB 0.0 249.4 47.0 202.4 234.3 31.8 0.0 5.0 107.1 100.6 2386 767 3.110 1147M 64M
Sum 626/9 37.72 GB 0.0 493.4 137.9 355.5 531.1 175.6 23.0 9.6 91.1 98.1 5547 3497 1.586 2247M 76M
Int 0/0 0.00 KB 0.0 0.3 0.0 0.3 0.5 0.3 0.0 2.0 64.9 129.7 4 10 0.405 1157K 1013
Uptime(secs): 986.6 total, 3.6 interval
Flush(GB): cumulative 55.401, interval 0.257
Cumulative compaction: 531.14 GB write, 551.30 MB/s write, 493.42 GB read, 512.15 MB/s read, 5546.7 seconds
Stalls(count): 118 level0_slowdown, 118 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 209 memtable_compaction, 0 memtable_slowdown, interval 0 total count
jobs=24
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 12/8 1.57 GB 1.0 71.5 0.0 71.5 126.7 55.3 0.0 2.3 63.2 112.0 1159 2141 0.541 322M 856K
L3 3/3 193.40 MB 0.0 22.4 14.4 8.1 22.2 14.1 0.0 1.5 104.0 102.9 221 14 15.778 101M 1042K
L4 5/0 322.34 MB 1.0 52.2 35.4 16.8 51.5 34.8 13.5 1.5 111.2 109.8 481 33 14.567 235M 2756K
L5 63/4 3.37 GB 1.0 99.1 40.8 58.3 97.7 39.4 11.3 2.4 63.2 62.3 1606 557 2.883 447M 6245K
L6 545/28 32.00 GB 0.0 265.0 47.5 217.5 249.5 32.0 0.0 5.3 96.4 90.8 2815 776 3.628 1219M 66M
Sum 628/43 37.45 GB 0.0 510.2 138.0 372.2 547.7 175.5 24.8 9.9 83.2 89.3 6281 3521 1.784 2325M 77M
Int 0/0 0.00 KB 0.0 8.9 1.3 7.6 8.6 1.0 0.0 24.4 89.4 86.6 102 35 2.911 40M 2787K
Uptime(secs): 995.4 total, 7.2 interval
Flush(GB): cumulative 55.401, interval 0.353
Cumulative compaction: 547.68 GB write, 563.42 MB/s write, 510.24 GB read, 524.90 MB/s read, 6281.2 seconds
Stalls(count): 75 level0_slowdown, 75 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 218 memtable_compaction, 0 memtable_slowdown, interval 9 total count
jobs=32
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 12/8 986.28 MB 1.0 68.9 0.0 68.9 124.2 55.3 0.0 2.2 61.5 110.8 1148 2142 0.536 310M 751K
L3 11/11 709.14 MB 0.0 23.6 14.7 9.0 23.4 14.4 0.0 1.6 103.6 102.5 234 15 15.578 106M 1095K
L4 5/0 311.04 MB 1.0 53.9 36.5 17.4 53.3 35.9 13.4 1.5 110.2 108.9 501 34 14.731 242M 2667K
L5 67/7 3.61 GB 1.0 100.5 41.2 59.3 99.1 39.8 11.0 2.4 61.7 60.9 1668 577 2.891 453M 6083K
L6 544/44 31.95 GB 0.0 272.1 47.4 224.7 256.6 32.0 0.0 5.4 92.6 87.3 3010 776 3.879 1252M 66M
Sum 639/70 37.52 GB 0.0 519.1 139.9 379.3 556.6 177.4 24.4 10.0 81.0 86.9 6560 3544 1.851 2366M 76M
Int 0/0 0.00 KB 0.0 1.4 0.5 0.9 1.5 0.6 0.0 8.0 36.7 41.4 38 15 2.527 6128K 77K
Uptime(secs): 1002.0 total, 5.5 interval
Flush(GB): cumulative 55.401, interval 0.193
Cumulative compaction: 556.64 GB write, 568.83 MB/s write, 519.12 GB read, 530.49 MB/s read, 6560.5 seconds
Stalls(count): 64 level0_slowdown, 64 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 302 memtable_compaction, 0 memtable_slowdown, interval 1 total count
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment