Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
<example-benchmark-fio-server-951-benchmark-9cd4p>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=36, BW=2354KiB/s (2411kB/s)(276MiB/120060msec)
slat (nsec): min=0, max=504592k, avg=441726.08, stdev=10395455.15
clat (nsec): min=0, max=995152k, avg=108771492.32, stdev=167527907.41
lat (nsec): min=0, max=995235k, avg=109214317.91, stdev=167771537.56
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1778],
| 30.00th=[ 2671], 40.00th=[ 6194], 50.00th=[ 34341], 60.00th=[ 77071],
| 70.00th=[107480], 80.00th=[158335], 90.00th=[358613], 95.00th=[501220],
| 99.00th=[759170], 99.50th=[868221], 99.90th=[935330], 99.95th=[935330],
| 99.99th=[994051]
bw ( KiB/s): min= 0, max= 3002, per=52.90%, avg=2352.50, stdev=918.53, samples=2
iops : min= 0, max= 46, avg=36.00, stdev=14.14, samples=2
lat (usec) : 4=0.02%, 500=0.02%, 750=0.07%, 1000=1.13%
lat (msec) : 2=22.09%, 4=12.19%, 10=7.36%, 20=4.44%, 50=4.51%
lat (msec) : 100=15.57%, 250=17.99%, 500=9.61%, 750=4.03%, 1000=1.02%
cpu : usr=0.04%, sys=0.16%, ctx=8445, majf=0, minf=31
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4413,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=348, BW=21.8MiB/s (22.8MB/s)(2614MiB/120030msec)
slat (nsec): min=0, max=498780k, avg=50078.56, stdev=3579794.07
clat (nsec): min=0, max=937417k, avg=11464988.97, stdev=59818482.14
lat (nsec): min=0, max=937452k, avg=11515884.91, stdev=59935424.53
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 764], 60.00th=[ 852], 70.00th=[ 924],
| 80.00th=[ 1288], 90.00th=[ 1941504], 95.00th=[ 51118080],
| 99.00th=[362807296], 99.50th=[471859200], 99.90th=[692060160],
| 99.95th=[734003200], 99.99th=[843055104]
bw ( KiB/s): min= 0, max=28792, per=91.76%, avg=22304.00, stdev=9175.42, samples=2
iops : min= 0, max= 450, avg=348.50, stdev=143.54, samples=2
lat (nsec) : 750=47.82%, 1000=25.51%
lat (usec) : 2=9.42%, 4=0.02%, 10=0.63%, 20=0.59%, 50=0.28%
lat (usec) : 100=0.10%, 250=0.04%, 500=0.29%, 750=1.01%, 1000=1.91%
lat (msec) : 2=2.44%, 4=1.35%, 10=0.63%, 20=0.48%, 50=2.41%
lat (msec) : 100=2.11%, 250=1.42%, 500=1.13%, 750=0.38%, 1000=0.03%
cpu : usr=0.11%, sys=0.37%, ctx=12986, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41881,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=31, BW=2014KiB/s (2062kB/s)(236MiB/120065msec)
slat (nsec): min=0, max=464438k, avg=251788.17, stdev=10003721.86
clat (nsec): min=0, max=761107k, avg=7488674.89, stdev=41769913.15
lat (nsec): min=0, max=761136k, avg=7741358.10, stdev=42918867.18
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 756], 30.00th=[ 844], 40.00th=[ 924],
| 50.00th=[ 1080], 60.00th=[ 1208], 70.00th=[ 1336],
| 80.00th=[ 1624], 90.00th=[ 1204224], 95.00th=[ 38535168],
| 99.00th=[214958080], 99.50th=[325058560], 99.90th=[534773760],
| 99.95th=[675282944], 99.99th=[759169024]
bw ( KiB/s): min= 0, max= 2421, per=8.29%, avg=2014.50, stdev=574.88, samples=2
iops : min= 0, max= 37, avg=31.00, stdev= 8.49, samples=2
write: IOPS=32, BW=2094KiB/s (2144kB/s)(246MiB/120065msec)
slat (nsec): min=0, max=552114k, avg=319489.42, stdev=8814876.07
clat (nsec): min=0, max=1106.9M, avg=115024253.69, stdev=172725727.36
lat (nsec): min=0, max=1107.2M, avg=115344697.54, stdev=172831915.20
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1205], 10.00th=[ 1401],
| 20.00th=[ 1876], 30.00th=[ 2900], 40.00th=[ 8029],
| 50.00th=[ 46924], 60.00th=[ 82314], 70.00th=[ 111674],
| 80.00th=[ 175113], 90.00th=[ 371196], 95.00th=[ 509608],
| 99.00th=[ 759170], 99.50th=[ 876610], 99.90th=[ 960496],
| 99.95th=[ 994051], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 2517, per=47.05%, avg=2092.50, stdev=600.33, samples=2
iops : min= 0, max= 39, avg=32.50, stdev= 9.19, samples=2
lat (nsec) : 750=9.36%, 1000=12.80%
lat (usec) : 2=17.65%, 4=0.09%, 10=0.25%, 20=0.70%, 50=0.53%
lat (usec) : 100=0.62%, 250=0.32%, 500=0.42%, 750=0.45%, 1000=1.00%
lat (msec) : 2=11.62%, 4=6.70%, 10=3.98%, 20=2.15%, 50=4.18%
lat (msec) : 100=9.19%, 250=9.53%, 500=5.70%, 750=2.26%, 1000=0.52%
cpu : usr=0.04%, sys=0.19%, ctx=8692, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3779,3925,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.7MiB/s (24.9MB/s), 2014KiB/s-21.8MiB/s (2062kB/s-22.8MB/s), io=2850MiB (2989MB), run=120030-120065msec
WRITE: bw=4448KiB/s (4554kB/s), 2094KiB/s-2354KiB/s (2144kB/s-2411kB/s), io=522MiB (547MB), run=120060-120065msec
Disk stats (read/write):
rbd300: ios=7336/8516, merge=0/1, ticks=525672/830529, in_queue=1348119, util=12.06%
Jobs: 117 (f=116): [_(2),m(1)][99.9%][r=1334M,w=800M][r=20.8k,w=12.5k IOPS][eta 00m:56s] <example-benchmark-fio-server-255-benchmark-v6glb>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=33, BW=2171KiB/s (2223kB/s)(254MiB/120036msec)
slat (nsec): min=0, max=536048k, avg=417781.11, stdev=10887713.42
clat (nsec): min=0, max=1261.6M, avg=118027748.44, stdev=178274062.80
lat (nsec): min=0, max=1261.6M, avg=118446560.37, stdev=178442905.58
clat percentiles (usec):
| 1.00th=[ 947], 5.00th=[ 1205], 10.00th=[ 1467],
| 20.00th=[ 2114], 30.00th=[ 3654], 40.00th=[ 10028],
| 50.00th=[ 58459], 60.00th=[ 86508], 70.00th=[ 111674],
| 80.00th=[ 177210], 90.00th=[ 371196], 95.00th=[ 530580],
| 99.00th=[ 775947], 99.50th=[ 901776], 99.90th=[1098908],
| 99.95th=[1098908], 99.99th=[1266680]
bw ( KiB/s): min= 0, max= 3001, per=53.65%, avg=2168.50, stdev=1177.33, samples=2
iops : min= 0, max= 46, avg=33.00, stdev=18.38, samples=2
lat (usec) : 750=0.05%, 1000=1.38%
lat (msec) : 2=17.77%, 4=11.92%, 10=8.85%, 20=3.91%, 50=4.47%
lat (msec) : 100=17.16%, 250=19.59%, 500=9.05%, 750=4.62%, 1000=0.96%
cpu : usr=0.04%, sys=0.15%, ctx=7612, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4068,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=273, BW=17.1MiB/s (17.9MB/s)(2052MiB/120023msec)
slat (nsec): min=0, max=393291k, avg=46338.23, stdev=2796058.39
clat (nsec): min=0, max=1038.1M, avg=14624047.78, stdev=65201130.73
lat (nsec): min=0, max=1038.6M, avg=14671240.74, stdev=65271622.43
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
| 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1144],
| 80.00th=[ 724992], 90.00th=[ 25559040], 95.00th=[ 67633152],
| 99.00th=[379584512], 99.50th=[488636416], 99.90th=[700448768],
| 99.95th=[750780416], 99.99th=[943718400]
bw ( KiB/s): min= 0, max=22634, per=90.78%, avg=17511.00, stdev=7245.02, samples=2
iops : min= 0, max= 354, avg=273.50, stdev=113.84, samples=2
lat (nsec) : 750=39.24%, 1000=25.97%
lat (usec) : 2=12.59%, 4=0.05%, 10=0.41%, 20=0.41%, 50=0.17%
lat (usec) : 100=0.09%, 250=0.03%, 500=0.24%, 750=1.12%, 1000=2.31%
lat (msec) : 2=2.99%, 4=1.91%, 10=0.98%, 20=0.94%, 50=3.84%
lat (msec) : 100=2.95%, 250=1.98%, 500=1.34%, 750=0.40%, 1000=0.05%
cpu : usr=0.10%, sys=0.33%, ctx=13746, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=32856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=27, BW=1786KiB/s (1829kB/s)(209MiB/120059msec)
slat (nsec): min=0, max=535574k, avg=187960.89, stdev=9251743.42
clat (nsec): min=0, max=794000k, avg=10453672.56, stdev=48478017.97
lat (nsec): min=0, max=794034k, avg=10642538.71, stdev=49327238.20
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 988],
| 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1448],
| 80.00th=[ 100864], 90.00th=[ 3194880], 95.00th=[ 53739520],
| 99.00th=[270532608], 99.50th=[362807296], 99.90th=[522190848],
| 99.95th=[608174080], 99.99th=[792723456]
bw ( KiB/s): min= 0, max= 2412, per=9.26%, avg=1786.50, stdev=884.59, samples=2
iops : min= 0, max= 37, avg=27.50, stdev=13.44, samples=2
write: IOPS=29, BW=1873KiB/s (1918kB/s)(220MiB/120059msec)
slat (nsec): min=0, max=400348k, avg=279434.69, stdev=6757705.47
clat (nsec): min=0, max=1261.4M, avg=126879957.42, stdev=186908714.01
lat (nsec): min=0, max=1261.5M, avg=127160436.98, stdev=186957434.42
clat percentiles (usec):
| 1.00th=[ 1004], 5.00th=[ 1287], 10.00th=[ 1549],
| 20.00th=[ 2278], 30.00th=[ 4228], 40.00th=[ 15795],
| 50.00th=[ 63701], 60.00th=[ 89654], 70.00th=[ 117965],
| 80.00th=[ 196084], 90.00th=[ 396362], 95.00th=[ 549454],
| 99.00th=[ 826278], 99.50th=[ 935330], 99.90th=[1098908],
| 99.95th=[1098908], 99.99th=[1266680]
bw ( KiB/s): min= 0, max= 2512, per=46.29%, avg=1871.00, stdev=906.51, samples=2
iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
lat (nsec) : 750=8.58%, 1000=11.38%
lat (usec) : 2=16.15%, 4=0.07%, 10=0.28%, 20=0.76%, 50=0.80%
lat (usec) : 100=1.02%, 250=0.67%, 500=0.63%, 750=0.63%, 1000=1.46%
lat (msec) : 2=9.58%, 4=7.00%, 10=4.52%, 20=2.42%, 50=3.92%
lat (msec) : 100=10.00%, 250=11.09%, 500=5.71%, 750=2.48%, 1000=0.71%
cpu : usr=0.04%, sys=0.17%, ctx=8048, majf=0, minf=33
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3351,3510,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=18.8MiB/s (19.8MB/s), 1786KiB/s-17.1MiB/s (1829kB/s-17.9MB/s), io=2262MiB (2372MB), run=120023-120059msec
WRITE: bw=4043KiB/s (4140kB/s), 1873KiB/s-2171KiB/s (1918kB/s-2223kB/s), io=474MiB (497MB), run=120036-120059msec
Disk stats (read/write):
rbd75: ios=8536/9131, merge=0/1, ticks=554901/875234, in_queue=1421062, util=12.57%
<example-benchmark-fio-server-424-benchmark-tlngh>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=41, BW=2666KiB/s (2730kB/s)(313MiB/120074msec)
slat (nsec): min=0, max=400546k, avg=326325.35, stdev=6957801.48
clat (nsec): min=0, max=1102.3M, avg=96092666.22, stdev=166659881.38
lat (nsec): min=0, max=1102.3M, avg=96420079.84, stdev=167148622.48
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1467],
| 20.00th=[ 1991], 30.00th=[ 2835], 40.00th=[ 4686],
| 50.00th=[ 11338], 60.00th=[ 45876], 70.00th=[ 86508],
| 80.00th=[ 133694], 90.00th=[ 346031], 95.00th=[ 480248],
| 99.00th=[ 801113], 99.50th=[ 843056], 99.90th=[ 943719],
| 99.95th=[ 985662], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 2965, per=55.11%, avg=2665.00, stdev=424.26, samples=2
iops : min= 0, max= 46, avg=41.00, stdev= 7.07, samples=2
lat (usec) : 500=0.02%, 750=0.04%, 1000=0.78%
lat (msec) : 2=19.24%, 4=16.86%, 10=12.00%, 20=6.18%, 50=5.52%
lat (msec) : 100=13.06%, 250=12.44%, 500=9.76%, 750=2.90%, 1000=1.22%
cpu : usr=0.05%, sys=0.18%, ctx=9369, majf=0, minf=31
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4999,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=318, BW=19.9MiB/s (20.9MB/s)(2389MiB/120031msec)
slat (nsec): min=0, max=320204k, avg=38057.84, stdev=2181510.66
clat (nsec): min=0, max=1057.4M, avg=12541947.82, stdev=61296772.33
lat (nsec): min=0, max=1057.4M, avg=12580866.82, stdev=61342276.53
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
| 50.00th=[ 852], 60.00th=[ 900], 70.00th=[ 1096],
| 80.00th=[ 43776], 90.00th=[ 6324224], 95.00th=[ 58982400],
| 99.00th=[362807296], 99.50th=[480247808], 99.90th=[700448768],
| 99.95th=[767557632], 99.99th=[893386752]
bw ( KiB/s): min= 0, max=25069, per=90.81%, avg=20383.00, stdev=6627.00, samples=2
iops : min= 0, max= 392, avg=318.50, stdev=103.94, samples=2
lat (nsec) : 750=33.09%, 1000=34.07%
lat (usec) : 2=11.70%, 4=0.05%, 10=0.57%, 20=0.41%, 50=0.14%
lat (usec) : 100=0.10%, 250=0.03%, 500=0.14%, 750=1.09%, 1000=2.45%
lat (msec) : 2=3.49%, 4=1.98%, 10=1.14%, 20=0.49%, 50=3.07%
lat (msec) : 100=2.94%, 250=1.55%, 500=1.06%, 750=0.39%, 1000=0.06%
cpu : usr=0.10%, sys=0.37%, ctx=15077, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=38240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=32, BW=2065KiB/s (2115kB/s)(242MiB/120022msec)
slat (nsec): min=0, max=399843k, avg=206891.70, stdev=7876972.64
clat (nsec): min=0, max=737486k, avg=8865298.83, stdev=46977819.17
lat (nsec): min=0, max=976153k, avg=9073210.18, stdev=48841859.71
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 772], 30.00th=[ 876], 40.00th=[ 972],
| 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1480],
| 80.00th=[ 109056], 90.00th=[ 2506752], 95.00th=[ 45875200],
| 99.00th=[244318208], 99.50th=[400556032], 99.90th=[583008256],
| 99.95th=[717225984], 99.99th=[734003200]
bw ( KiB/s): min= 0, max= 2312, per=9.20%, avg=2065.00, stdev=349.31, samples=2
iops : min= 0, max= 36, avg=32.00, stdev= 5.66, samples=2
write: IOPS=33, BW=2171KiB/s (2223kB/s)(255MiB/120022msec)
slat (nsec): min=0, max=10291k, avg=180936.82, stdev=303505.98
clat (nsec): min=0, max=1040.1M, avg=109529981.00, stdev=175834302.36
lat (nsec): min=0, max=1040.2M, avg=109712031.36, stdev=175783939.65
clat percentiles (usec):
| 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1565],
| 20.00th=[ 2278], 30.00th=[ 3621], 40.00th=[ 6587],
| 50.00th=[ 19530], 60.00th=[ 66847], 70.00th=[ 102237],
| 80.00th=[ 160433], 90.00th=[ 371196], 95.00th=[ 497026],
| 99.00th=[ 809501], 99.50th=[ 893387], 99.90th=[ 952108],
| 99.95th=[1002439], 99.99th=[1035994]
bw ( KiB/s): min= 0, max= 2405, per=44.85%, avg=2169.00, stdev=333.75, samples=2
iops : min= 0, max= 37, avg=33.50, stdev= 4.95, samples=2
lat (nsec) : 750=8.20%, 1000=12.22%
lat (usec) : 2=14.93%, 4=0.05%, 10=0.39%, 20=1.15%, 50=1.02%
lat (usec) : 100=0.96%, 250=0.79%, 500=0.94%, 750=0.52%, 1000=1.17%
lat (msec) : 2=9.61%, 4=8.78%, 10=6.61%, 20=3.36%, 50=4.33%
lat (msec) : 100=8.36%, 250=7.93%, 500=6.17%, 750=1.85%, 1000=0.68%
cpu : usr=0.05%, sys=0.19%, ctx=9240, majf=0, minf=35
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3874,4069,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=21.9MiB/s (22.0MB/s), 2065KiB/s-19.9MiB/s (2115kB/s-20.9MB/s), io=2631MiB (2759MB), run=120022-120031msec
WRITE: bw=4836KiB/s (4953kB/s), 2171KiB/s-2666KiB/s (2223kB/s-2730kB/s), io=567MiB (595MB), run=120022-120074msec
Disk stats (read/write):
rbd121: ios=8410/9542, merge=0/1, ticks=542895/868442, in_queue=1402193, util=12.62%
<example-benchmark-fio-server-325-benchmark-fbmfd>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=46, BW=2969KiB/s (3040kB/s)(348MiB/120063msec)
slat (nsec): min=0, max=428628k, avg=322718.17, stdev=6988005.94
clat (nsec): min=0, max=1469.7M, avg=86229567.91, stdev=147784132.72
lat (nsec): min=0, max=1469.8M, avg=86553448.19, stdev=148051015.99
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1401],
| 20.00th=[ 1827], 30.00th=[ 2573], 40.00th=[ 4359],
| 50.00th=[ 10814], 60.00th=[ 61604], 70.00th=[ 91751],
| 80.00th=[ 120062], 90.00th=[ 261096], 95.00th=[ 417334],
| 99.00th=[ 683672], 99.50th=[ 809501], 99.90th=[ 977273],
| 99.95th=[1283458], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 3621, per=54.56%, avg=2967.50, stdev=924.19, samples=2
iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
lat (usec) : 50=0.02%, 250=0.02%, 750=0.07%, 1000=1.02%
lat (msec) : 2=21.88%, 4=15.52%, 10=11.03%, 20=4.01%, 50=3.92%
lat (msec) : 100=15.99%, 250=16.39%, 500=6.95%, 750=2.48%, 1000=0.68%
cpu : usr=0.07%, sys=0.19%, ctx=10557, majf=0, minf=31
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5566,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=295, BW=18.5MiB/s (19.4MB/s)(2218MiB/120022msec)
slat (nsec): min=0, max=413228k, avg=42515.07, stdev=2589450.95
clat (nsec): min=0, max=1400.0M, avg=13517573.15, stdev=63183262.34
lat (nsec): min=0, max=1400.3M, avg=13560936.96, stdev=63244141.80
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
| 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 772],
| 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1080],
| 80.00th=[ 264192], 90.00th=[ 22937600], 95.00th=[ 61079552],
| 99.00th=[375390208], 99.50th=[480247808], 99.90th=[700448768],
| 99.95th=[767557632], 99.99th=[926941184]
bw ( KiB/s): min= 0, max=24299, per=88.89%, avg=18927.00, stdev=7597.16, samples=2
iops : min= 0, max= 380, avg=295.50, stdev=119.50, samples=2
lat (nsec) : 750=36.46%, 1000=30.96%
lat (usec) : 2=11.74%, 4=0.06%, 10=0.34%, 20=0.18%, 50=0.17%
lat (usec) : 100=0.06%, 250=0.05%, 500=0.17%, 750=1.13%, 1000=2.16%
lat (msec) : 2=2.93%, 4=1.72%, 10=1.00%, 20=0.66%, 50=3.78%
lat (msec) : 100=3.29%, 250=1.50%, 500=1.19%, 750=0.41%, 1000=0.06%
cpu : usr=0.10%, sys=0.35%, ctx=14216, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=35512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=37, BW=2372KiB/s (2429kB/s)(278MiB/120051msec)
slat (nsec): min=0, max=405259k, avg=173230.37, stdev=7000712.24
clat (nsec): min=0, max=676905k, avg=10063171.52, stdev=47810644.50
lat (nsec): min=0, max=676931k, avg=10237323.34, stdev=48310601.69
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 812], 30.00th=[ 876], 40.00th=[ 980],
| 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1416],
| 80.00th=[ 73216], 90.00th=[ 3588096], 95.00th=[ 55836672],
| 99.00th=[265289728], 99.50th=[362807296], 99.90th=[583008256],
| 99.95th=[599785472], 99.99th=[675282944]
bw ( KiB/s): min= 0, max= 2913, per=11.14%, avg=2373.00, stdev=763.68, samples=2
iops : min= 0, max= 45, avg=36.50, stdev=12.02, samples=2
write: IOPS=38, BW=2471KiB/s (2531kB/s)(290MiB/120051msec)
slat (nsec): min=0, max=4214.0k, avg=178472.88, stdev=241439.55
clat (nsec): min=0, max=1469.7M, avg=93902576.19, stdev=154811764.20
lat (nsec): min=0, max=1469.8M, avg=94082275.96, stdev=154763959.10
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1450],
| 20.00th=[ 1991], 30.00th=[ 3097], 40.00th=[ 5604],
| 50.00th=[ 22152], 60.00th=[ 70779], 70.00th=[ 95945],
| 80.00th=[ 129500], 90.00th=[ 291505], 95.00th=[ 442500],
| 99.00th=[ 750781], 99.50th=[ 876610], 99.90th=[1098908],
| 99.95th=[1132463], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 3003, per=45.41%, avg=2470.00, stdev=753.78, samples=2
iops : min= 0, max= 46, avg=38.00, stdev=11.31, samples=2
lat (nsec) : 750=6.72%, 1000=13.63%
lat (usec) : 2=17.22%, 4=0.02%, 10=0.23%, 20=0.47%, 50=0.44%
lat (usec) : 100=0.66%, 250=0.53%, 500=0.37%, 750=0.68%, 1000=1.45%
lat (msec) : 2=11.21%, 4=8.28%, 10=5.91%, 20=2.37%, 50=3.56%
lat (msec) : 100=10.54%, 250=9.54%, 500=4.41%, 750=1.28%, 1000=0.43%
cpu : usr=0.07%, sys=0.22%, ctx=10658, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4451,4633,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=20.8MiB/s (21.8MB/s), 2372KiB/s-18.5MiB/s (2429kB/s-19.4MB/s), io=2496MiB (2618MB), run=120022-120051msec
WRITE: bw=5440KiB/s (5570kB/s), 2471KiB/s-2969KiB/s (2531kB/s-3040kB/s), io=638MiB (669MB), run=120051-120063msec
Disk stats (read/write):
rbd103: ios=8316/10505, merge=0/1, ticks=549459/828531, in_queue=1368379, util=14.25%
<example-benchmark-fio-server-336-benchmark-tzn4v>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=37, BW=2378KiB/s (2435kB/s)(279MiB/120104msec)
slat (nsec): min=0, max=340097k, avg=326004.44, stdev=6645041.68
clat (nsec): min=0, max=1280.6M, avg=107678890.86, stdev=175225797.88
lat (nsec): min=0, max=1280.7M, avg=108005935.42, stdev=175238266.51
clat percentiles (usec):
| 1.00th=[ 955], 5.00th=[ 1172], 10.00th=[ 1369],
| 20.00th=[ 1844], 30.00th=[ 2868], 40.00th=[ 6128],
| 50.00th=[ 28967], 60.00th=[ 76022], 70.00th=[ 104334],
| 80.00th=[ 154141], 90.00th=[ 337642], 95.00th=[ 488637],
| 99.00th=[ 817890], 99.50th=[ 935330], 99.90th=[1132463],
| 99.95th=[1283458], 99.99th=[1283458]
bw ( KiB/s): min= 0, max= 1784, per=40.93%, avg=1784.00, stdev= 0.00, samples=1
iops : min= 0, max= 27, avg=27.00, stdev= 0.00, samples=1
lat (usec) : 4=0.02%, 100=0.02%, 250=0.02%, 500=0.02%, 750=0.09%
lat (usec) : 1000=1.68%
lat (msec) : 2=20.36%, 4=13.09%, 10=8.88%, 20=3.61%, 50=5.94%
lat (msec) : 100=14.64%, 250=17.60%, 500=9.35%, 750=3.27%, 1000=1.03%
cpu : usr=0.04%, sys=0.17%, ctx=8470, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4460,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=456, BW=28.5MiB/s (29.9MB/s)(3422MiB/120027msec)
slat (nsec): min=0, max=338266k, avg=27425.06, stdev=1786682.20
clat (nsec): min=0, max=999544k, avg=8748171.03, stdev=51624016.82
lat (nsec): min=0, max=999606k, avg=8776407.06, stdev=51663169.95
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
| 50.00th=[ 748], 60.00th=[ 828], 70.00th=[ 884],
| 80.00th=[ 1112], 90.00th=[ 1044480], 95.00th=[ 36962304],
| 99.00th=[270532608], 99.50th=[417333248], 99.90th=[650117120],
| 99.95th=[742391808], 99.99th=[910163968]
bw ( KiB/s): min= 0, max=44025, per=93.94%, avg=29198.00, stdev=20968.54, samples=2
iops : min= 0, max= 688, avg=456.00, stdev=328.10, samples=2
lat (nsec) : 750=50.41%, 1000=27.05%
lat (usec) : 2=8.04%, 4=0.05%, 10=1.12%, 20=0.45%, 50=0.17%
lat (usec) : 100=0.10%, 250=0.06%, 500=0.31%, 750=0.70%, 1000=1.39%
lat (msec) : 2=1.85%, 4=1.13%, 10=0.65%, 20=0.46%, 50=2.03%
lat (msec) : 100=1.78%, 250=1.14%, 500=0.82%, 750=0.24%, 1000=0.05%
cpu : usr=0.12%, sys=0.47%, ctx=13690, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=54795,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=29, BW=1895KiB/s (1941kB/s)(222MiB/120051msec)
slat (nsec): min=0, max=263179k, avg=101624.02, stdev=4414051.05
clat (nsec): min=0, max=655890k, avg=9618104.13, stdev=46649931.58
lat (nsec): min=0, max=655927k, avg=9720798.57, stdev=46853921.91
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 804], 30.00th=[ 876], 40.00th=[ 956],
| 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1416],
| 80.00th=[ 35584], 90.00th=[ 2670592], 95.00th=[ 50069504],
| 99.00th=[256901120], 99.50th=[367001600], 99.90th=[522190848],
| 99.95th=[599785472], 99.99th=[658505728]
bw ( KiB/s): min= 0, max= 2421, per=6.10%, avg=1895.50, stdev=743.17, samples=2
iops : min= 0, max= 37, avg=29.00, stdev=11.31, samples=2
write: IOPS=30, BW=1983KiB/s (2030kB/s)(232MiB/120051msec)
slat (nsec): min=0, max=279271k, avg=268855.21, stdev=4731857.77
clat (nsec): min=0, max=1280.7M, avg=120132593.80, stdev=184042868.06
lat (nsec): min=0, max=1280.8M, avg=120402554.96, stdev=184195357.79
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1221], 10.00th=[ 1450],
| 20.00th=[ 2057], 30.00th=[ 3654], 40.00th=[ 10159],
| 50.00th=[ 51643], 60.00th=[ 86508], 70.00th=[ 114820],
| 80.00th=[ 187696], 90.00th=[ 367002], 95.00th=[ 505414],
| 99.00th=[ 876610], 99.50th=[ 977273], 99.90th=[1132463],
| 99.95th=[1132463], 99.99th=[1283458]
bw ( KiB/s): min= 0, max= 2540, per=45.43%, avg=1980.50, stdev=791.25, samples=2
iops : min= 0, max= 39, avg=30.50, stdev=12.02, samples=2
lat (nsec) : 750=6.97%, 1000=14.22%
lat (usec) : 2=16.17%, 4=0.08%, 10=0.36%, 20=0.89%, 50=0.76%
lat (usec) : 100=0.98%, 250=0.47%, 500=0.58%, 750=0.58%, 1000=1.20%
lat (msec) : 2=10.26%, 4=6.75%, 10=4.73%, 20=1.90%, 50=4.94%
lat (msec) : 100=8.73%, 250=10.95%, 500=5.79%, 750=1.76%, 1000=0.80%
cpu : usr=0.05%, sys=0.18%, ctx=8332, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3556,3716,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=30.4MiB/s (31.8MB/s), 1895KiB/s-28.5MiB/s (1941kB/s-29.9MB/s), io=3644MiB (3821MB), run=120027-120051msec
WRITE: bw=4360KiB/s (4465kB/s), 1983KiB/s-2378KiB/s (2030kB/s-2435kB/s), io=511MiB (536MB), run=120051-120104msec
Disk stats (read/write):
rbd111: ios=7834/8600, merge=0/1, ticks=524255/818436, in_queue=1334384, util=12.21%
fio: inflate error -5
fio: failed decompressing log
fio: failed converting IO log
<example-benchmark-fio-server-239-benchmark-4zbns>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=45, BW=2936KiB/s (3006kB/s)(344MiB/120067msec)
slat (nsec): min=0, max=534902k, avg=347032.55, stdev=8533504.59
clat (nsec): min=0, max=889389k, avg=86828735.19, stdev=144120444.95
lat (nsec): min=0, max=1368.6M, avg=87177012.08, stdev=145119573.59
clat percentiles (usec):
| 1.00th=[ 947], 5.00th=[ 1156], 10.00th=[ 1336], 20.00th=[ 1795],
| 30.00th=[ 2606], 40.00th=[ 4948], 50.00th=[ 17433], 60.00th=[ 55313],
| 70.00th=[ 94897], 80.00th=[131597], 90.00th=[238027], 95.00th=[421528],
| 99.00th=[700449], 99.50th=[742392], 99.90th=[826278], 99.95th=[859833],
| 99.99th=[893387]
bw ( KiB/s): min= 0, max= 3771, per=55.94%, avg=2935.00, stdev=1182.28, samples=2
iops : min= 0, max= 58, avg=45.00, stdev=18.38, samples=2
lat (usec) : 250=0.02%, 500=0.04%, 750=0.02%, 1000=1.76%
lat (msec) : 2=21.13%, 4=14.35%, 10=9.14%, 20=4.60%, 50=7.45%
lat (msec) : 100=13.01%, 250=18.91%, 500=6.21%, 750=2.98%, 1000=0.45%
cpu : usr=0.05%, sys=0.20%, ctx=10277, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5505,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=255, BW=15.9MiB/s (16.7MB/s)(1912MiB/120021msec)
slat (nsec): min=0, max=533552k, avg=48056.18, stdev=3101874.04
clat (nsec): min=0, max=925627k, avg=15680255.74, stdev=65921046.09
lat (nsec): min=0, max=925689k, avg=15729185.26, stdev=66004552.52
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
| 50.00th=[ 868], 60.00th=[ 956], 70.00th=[ 1256],
| 80.00th=[ 970752], 90.00th=[ 31850496], 95.00th=[ 73924608],
| 99.00th=[375390208], 99.50th=[497025024], 99.90th=[692060160],
| 99.95th=[725614592], 99.99th=[834666496]
bw ( KiB/s): min= 0, max=22069, per=100.00%, avg=22069.00, stdev= 0.00, samples=1
iops : min= 0, max= 345, avg=345.00, stdev= 0.00, samples=1
lat (nsec) : 750=37.72%, 1000=24.33%
lat (usec) : 2=12.99%, 4=0.07%, 10=0.30%, 20=0.28%, 50=0.21%
lat (usec) : 100=0.03%, 250=0.04%, 500=0.32%, 750=1.28%, 1000=2.65%
lat (msec) : 2=3.61%, 4=2.23%, 10=1.10%, 20=1.05%, 50=4.52%
lat (msec) : 100=3.20%, 250=2.16%, 500=1.44%, 750=0.45%, 1000=0.03%
cpu : usr=0.08%, sys=0.33%, ctx=14593, majf=0, minf=100
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=30613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120027msec)
slat (nsec): min=0, max=534280k, avg=237292.72, stdev=9787518.53
clat (nsec): min=0, max=711810k, avg=12200922.95, stdev=52573178.22
lat (nsec): min=0, max=711830k, avg=12439097.46, stdev=53435173.79
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 988],
| 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1528],
| 80.00th=[ 238592], 90.00th=[ 20840448], 95.00th=[ 67633152],
| 99.00th=[283115520], 99.50th=[358612992], 99.90th=[675282944],
| 99.95th=[708837376], 99.99th=[708837376]
bw ( KiB/s): min= 0, max= 2634, per=11.93%, avg=2210.00, stdev=599.63, samples=2
iops : min= 0, max= 41, avg=34.00, stdev= 9.90, samples=2
write: IOPS=36, BW=2313KiB/s (2368kB/s)(271MiB/120027msec)
slat (nsec): min=0, max=6942.8k, avg=174147.77, stdev=263171.86
clat (nsec): min=0, max=859436k, avg=98737609.79, stdev=152398197.87
lat (nsec): min=0, max=859576k, avg=98912930.74, stdev=152347644.57
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1303], 10.00th=[ 1549], 20.00th=[ 2180],
| 30.00th=[ 3687], 40.00th=[ 10159], 50.00th=[ 39584], 60.00th=[ 73925],
| 70.00th=[105382], 80.00th=[141558], 90.00th=[295699], 95.00th=[463471],
| 99.00th=[692061], 99.50th=[750781], 99.90th=[801113], 99.95th=[843056],
| 99.99th=[859833]
bw ( KiB/s): min= 0, max= 2733, per=44.05%, avg=2311.50, stdev=596.09, samples=2
iops : min= 0, max= 42, avg=35.50, stdev= 9.19, samples=2
lat (nsec) : 750=7.59%, 1000=12.44%
lat (usec) : 2=15.02%, 4=0.05%, 10=0.29%, 20=1.04%, 50=0.85%
lat (usec) : 100=0.93%, 250=0.97%, 500=0.55%, 750=0.57%, 1000=1.29%
lat (msec) : 2=9.95%, 4=7.29%, 10=5.11%, 20=3.11%, 50=5.72%
lat (msec) : 100=9.42%, 250=11.52%, 500=3.96%, 750=2.13%, 1000=0.22%
cpu : usr=0.05%, sys=0.20%, ctx=9894, majf=0, minf=36
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4146,4334,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=18.1MiB/s (18.0MB/s), 2211KiB/s-15.9MiB/s (2264kB/s-16.7MB/s), io=2171MiB (2277MB), run=120021-120027msec
WRITE: bw=5248KiB/s (5374kB/s), 2313KiB/s-2936KiB/s (2368kB/s-3006kB/s), io=615MiB (645MB), run=120027-120067msec
Disk stats (read/write):
rbd95: ios=9683/11004, merge=0/1, ticks=566653/848675, in_queue=1404758, util=14.29%
<example-benchmark-fio-server-747-benchmark-jh49p>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=41, BW=2650KiB/s (2714kB/s)(311MiB/120014msec)
slat (nsec): min=0, max=444994k, avg=406213.93, stdev=8650946.30
clat (nsec): min=0, max=1306.6M, avg=96754907.14, stdev=155893560.13
lat (nsec): min=0, max=1306.7M, avg=97162272.04, stdev=156021316.52
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1352],
| 20.00th=[ 1811], 30.00th=[ 2704], 40.00th=[ 5473],
| 50.00th=[ 35914], 60.00th=[ 78119], 70.00th=[ 100140],
| 80.00th=[ 130548], 90.00th=[ 278922], 95.00th=[ 471860],
| 99.00th=[ 717226], 99.50th=[ 817890], 99.90th=[1044382],
| 99.95th=[1098908], 99.99th=[1300235]
bw ( KiB/s): min= 0, max= 3854, per=54.00%, avg=2648.00, stdev=1705.54, samples=2
iops : min= 0, max= 60, avg=41.00, stdev=26.87, samples=2
lat (usec) : 500=0.04%, 750=0.02%, 1000=1.43%
lat (msec) : 2=21.20%, 4=14.15%, 10=7.79%, 20=3.60%, 50=3.85%
lat (msec) : 100=17.74%, 250=19.33%, 500=6.83%, 750=3.32%, 1000=0.62%
cpu : usr=0.04%, sys=0.18%, ctx=9502, majf=0, minf=29
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4967,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=565, BW=35.3MiB/s (36.0MB/s)(4233MiB/120022msec)
slat (nsec): min=0, max=426774k, avg=30081.64, stdev=2287324.85
clat (nsec): min=0, max=1057.1M, avg=7035019.24, stdev=47212773.76
lat (nsec): min=0, max=1057.1M, avg=7065919.65, stdev=47275485.63
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 716], 30.00th=[ 724], 40.00th=[ 740],
| 50.00th=[ 796], 60.00th=[ 852], 70.00th=[ 900],
| 80.00th=[ 1112], 90.00th=[ 25216], 95.00th=[ 13959168],
| 99.00th=[227540992], 99.50th=[396361728], 99.90th=[633339904],
| 99.95th=[725614592], 99.99th=[918552576]
bw ( KiB/s): min= 0, max=54834, per=94.38%, avg=36118.50, stdev=26467.71, samples=2
iops : min= 0, max= 860, avg=565.50, stdev=416.49, samples=2
lat (nsec) : 750=45.37%, 1000=32.21%
lat (usec) : 2=6.94%, 4=0.05%, 10=2.83%, 20=2.39%, 50=0.47%
lat (usec) : 100=0.13%, 250=0.04%, 500=0.09%, 750=0.50%, 1000=1.06%
lat (msec) : 2=1.45%, 4=0.88%, 10=0.46%, 20=0.29%, 50=1.68%
lat (msec) : 100=1.44%, 250=0.80%, 500=0.68%, 750=0.19%, 1000=0.04%
cpu : usr=0.16%, sys=0.52%, ctx=13040, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=67920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=33, BW=2161KiB/s (2213kB/s)(253MiB/120042msec)
slat (nsec): min=0, max=4021.6k, avg=26307.06, stdev=106378.46
clat (nsec): min=0, max=626922k, avg=7607568.80, stdev=38180756.57
lat (nsec): min=0, max=626958k, avg=7634781.88, stdev=38187137.32
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
| 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1384],
| 80.00th=[ 17024], 90.00th=[ 1581056], 95.00th=[ 47448064],
| 99.00th=[191889408], 99.50th=[304087040], 99.90th=[480247808],
| 99.95th=[484442112], 99.99th=[624951296]
bw ( KiB/s): min= 0, max= 2935, per=5.65%, avg=2161.50, stdev=1093.89, samples=2
iops : min= 0, max= 45, avg=33.00, stdev=16.97, samples=2
write: IOPS=35, BW=2255KiB/s (2309kB/s)(264MiB/120042msec)
slat (nsec): min=0, max=444199k, avg=469547.59, stdev=10265636.14
clat (nsec): min=0, max=1282.2M, avg=105745971.30, stdev=159456781.24
lat (nsec): min=0, max=1282.3M, avg=106216461.29, stdev=159801960.04
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
| 20.00th=[ 1909], 30.00th=[ 3228], 40.00th=[ 9503],
| 50.00th=[ 60031], 60.00th=[ 86508], 70.00th=[ 107480],
| 80.00th=[ 139461], 90.00th=[ 320865], 95.00th=[ 484443],
| 99.00th=[ 734004], 99.50th=[ 801113], 99.90th=[ 943719],
| 99.95th=[1027605], 99.99th=[1283458]
bw ( KiB/s): min= 0, max= 3069, per=45.93%, avg=2252.50, stdev=1154.71, samples=2
iops : min= 0, max= 47, avg=34.50, stdev=17.68, samples=2
lat (nsec) : 750=8.79%, 1000=13.03%
lat (usec) : 2=16.53%, 4=0.04%, 10=0.41%, 20=0.77%, 50=0.72%
lat (usec) : 100=0.92%, 250=0.56%, 500=0.41%, 750=0.45%, 1000=1.28%
lat (msec) : 2=11.07%, 4=6.54%, 10=4.13%, 20=1.80%, 50=3.83%
lat (msec) : 100=10.84%, 250=11.47%, 500=4.07%, 750=1.91%, 1000=0.43%
cpu : usr=0.05%, sys=0.21%, ctx=9390, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4056,4226,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=37.4MiB/s (39.2MB/s), 2161KiB/s-35.3MiB/s (2213kB/s-36.0MB/s), io=4486MiB (4704MB), run=120022-120042msec
WRITE: bw=4904KiB/s (5022kB/s), 2255KiB/s-2650KiB/s (2309kB/s-2714kB/s), io=575MiB (603MB), run=120014-120042msec
Disk stats (read/write):
rbd244: ios=7107/9294, merge=0/3, ticks=515812/787469, in_queue=1295011, util=12.37%
<example-benchmark-fio-server-238-benchmark-wczfn>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=39, BW=2517KiB/s (2577kB/s)(295MiB/120014msec)
slat (nsec): min=0, max=341058k, avg=292693.61, stdev=5575637.86
clat (nsec): min=0, max=1342.0M, avg=102021795.41, stdev=158984969.89
lat (nsec): min=0, max=1342.1M, avg=102315643.79, stdev=158993907.50
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1483],
| 20.00th=[ 2073], 30.00th=[ 3392], 40.00th=[ 9372],
| 50.00th=[ 45351], 60.00th=[ 78119], 70.00th=[ 106431],
| 80.00th=[ 143655], 90.00th=[ 295699], 95.00th=[ 459277],
| 99.00th=[ 725615], 99.50th=[ 868221], 99.90th=[1149240],
| 99.95th=[1300235], 99.99th=[1333789]
bw ( KiB/s): min= 0, max= 3384, per=53.24%, avg=2514.50, stdev=1229.66, samples=2
iops : min= 0, max= 52, avg=38.50, stdev=19.09, samples=2
lat (usec) : 50=0.02%, 250=0.02%, 500=0.06%, 750=0.02%, 1000=1.34%
lat (msec) : 2=17.45%, 4=13.19%, 10=8.25%, 20=3.90%, 50=7.19%
lat (msec) : 100=16.22%, 250=20.54%, 500=7.70%, 750=3.41%, 1000=0.49%
cpu : usr=0.05%, sys=0.17%, ctx=8969, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4717,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=379, BW=23.7MiB/s (24.9MB/s)(2846MiB/120038msec)
slat (nsec): min=0, max=313498k, avg=28520.52, stdev=1565230.47
clat (nsec): min=0, max=1558.7M, avg=10539488.90, stdev=56101509.52
lat (nsec): min=0, max=1558.8M, avg=10568843.17, stdev=56135662.02
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
| 50.00th=[ 764], 60.00th=[ 860], 70.00th=[ 924],
| 80.00th=[ 1288], 90.00th=[ 2441216], 95.00th=[ 49020928],
| 99.00th=[316669952], 99.50th=[450887680], 99.90th=[666894336],
| 99.95th=[734003200], 99.99th=[977272832]
bw ( KiB/s): min= 0, max=34871, per=92.05%, avg=24286.50, stdev=14968.74, samples=2
iops : min= 0, max= 545, avg=379.50, stdev=234.05, samples=2
lat (nsec) : 750=47.29%, 1000=26.10%
lat (usec) : 2=9.46%, 4=0.05%, 10=1.23%, 20=0.46%, 50=0.15%
lat (usec) : 100=0.07%, 250=0.06%, 500=0.26%, 750=0.85%, 1000=1.47%
lat (msec) : 2=2.07%, 4=1.36%, 10=0.71%, 20=0.54%, 50=2.97%
lat (msec) : 100=2.30%, 250=1.30%, 500=0.98%, 750=0.30%, 1000=0.03%
cpu : usr=0.12%, sys=0.40%, ctx=13768, majf=0, minf=42
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=45580,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=32, BW=2105KiB/s (2155kB/s)(247MiB/120008msec)
slat (nsec): min=0, max=267135k, avg=128561.10, stdev=4765559.54
clat (nsec): min=0, max=638238k, avg=9563016.49, stdev=43149253.15
lat (nsec): min=0, max=638271k, avg=9692556.97, stdev=43396123.16
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 996],
| 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
| 80.00th=[ 40192], 90.00th=[ 3751936], 95.00th=[ 54788096],
| 99.00th=[254803968], 99.50th=[333447168], 99.90th=[497025024],
| 99.95th=[509607936], 99.99th=[641728512]
bw ( KiB/s): min= 0, max= 2695, per=7.97%, avg=2104.00, stdev=835.80, samples=2
iops : min= 0, max= 42, avg=32.50, stdev=13.44, samples=2
write: IOPS=34, BW=2206KiB/s (2259kB/s)(259MiB/120008msec)
slat (nsec): min=0, max=10176k, avg=175306.89, stdev=326702.12
clat (nsec): min=0, max=1839.1M, avg=107317664.14, stdev=164884666.88
lat (nsec): min=0, max=1839.2M, avg=107494142.40, stdev=164838936.04
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1467],
| 20.00th=[ 2212], 30.00th=[ 3949], 40.00th=[ 12256],
| 50.00th=[ 52167], 60.00th=[ 81265], 70.00th=[ 108528],
| 80.00th=[ 154141], 90.00th=[ 312476], 95.00th=[ 484443],
| 99.00th=[ 734004], 99.50th=[ 876610], 99.90th=[1149240],
| 99.95th=[1300235], 99.99th=[1837106]
bw ( KiB/s): min= 0, max= 2809, per=46.68%, avg=2204.50, stdev=854.89, samples=2
iops : min= 0, max= 43, avg=34.00, stdev=12.73, samples=2
lat (nsec) : 750=7.67%, 1000=12.10%
lat (usec) : 2=17.50%, 4=0.09%, 10=0.22%, 20=0.95%, 50=0.78%
lat (usec) : 100=1.11%, 250=0.52%, 500=0.40%, 750=0.49%, 1000=1.18%
lat (msec) : 2=9.58%, 4=6.85%, 10=4.79%, 20=2.12%, 50=5.21%
lat (msec) : 100=10.05%, 250=11.35%, 500=4.74%, 750=1.89%, 1000=0.30%
cpu : usr=0.06%, sys=0.20%, ctx=9317, majf=0, minf=37
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3948,4134,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=25.8MiB/s (27.0MB/s), 2105KiB/s-23.7MiB/s (2155kB/s-24.9MB/s), io=3093MiB (3243MB), run=120008-120038msec
WRITE: bw=4723KiB/s (4837kB/s), 2206KiB/s-2517KiB/s (2259kB/s-2577kB/s), io=554MiB (580MB), run=120008-120014msec
Disk stats (read/write):
rbd67: ios=8366/9287, merge=0/1, ticks=537421/816981, in_queue=1345358, util=13.52%
<example-benchmark-fio-server-965-benchmark-cvg7r>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
write: IOPS=46, BW=2948KiB/s (3018kB/s)(346MiB/120088msec)
slat (nsec): min=0, max=479406k, avg=350130.57, stdev=8587720.43
clat (nsec): min=0, max=1103.4M, avg=86740940.20, stdev=143041701.87
lat (nsec): min=0, max=1103.5M, avg=87092248.17, stdev=143156153.60
clat percentiles (usec):
| 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1352],
| 20.00th=[ 1795], 30.00th=[ 2737], 40.00th=[ 5735],
| 50.00th=[ 20317], 60.00th=[ 67634], 70.00th=[ 96994],
| 80.00th=[ 129500], 90.00th=[ 229639], 95.00th=[ 387974],
| 99.00th=[ 700449], 99.50th=[ 801113], 99.90th=[1044382],
| 99.95th=[1098908], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 3585, per=54.32%, avg=2947.50, stdev=901.56, samples=2
iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
lat (usec) : 4=0.02%, 500=0.02%, 750=0.02%, 1000=1.74%
lat (msec) : 2=21.58%, 4=12.23%, 10=9.90%, 20=4.40%, 50=5.70%
lat (msec) : 100=15.77%, 250=19.52%, 500=6.10%, 750=2.35%, 1000=0.52%
cpu : usr=0.07%, sys=0.19%, ctx=10526, majf=0, minf=26
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5528,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
read: IOPS=291, BW=18.2MiB/s (19.1MB/s)(2187MiB/120030msec)
slat (nsec): min=0, max=467384k, avg=48463.95, stdev=3199541.13
clat (nsec): min=0, max=987567k, avg=13728849.56, stdev=63522045.77
lat (nsec): min=0, max=987595k, avg=13778150.67, stdev=63621954.04
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
| 50.00th=[ 820], 60.00th=[ 884], 70.00th=[ 1096],
| 80.00th=[ 522240], 90.00th=[ 20840448], 95.00th=[ 63700992],
| 99.00th=[371195904], 99.50th=[492830720], 99.90th=[683671552],
| 99.95th=[750780416], 99.99th=[851443712]
bw ( KiB/s): min= 0, max=24862, per=88.70%, avg=18656.00, stdev=8776.61, samples=2
iops : min= 0, max= 388, avg=291.00, stdev=137.18, samples=2
lat (nsec) : 750=42.00%, 1000=25.39%
lat (usec) : 2=11.31%, 4=0.08%, 10=0.45%, 20=0.12%, 50=0.11%
lat (usec) : 100=0.05%, 250=0.06%, 500=0.44%, 750=1.18%, 1000=2.35%
lat (msec) : 2=3.08%, 4=1.93%, 10=0.88%, 20=0.54%, 50=3.67%
lat (msec) : 100=3.09%, 250=1.55%, 500=1.29%, 750=0.44%, 1000=0.05%
cpu : usr=0.10%, sys=0.34%, ctx=14233, majf=0, minf=74
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=35008,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
read: IOPS=37, BW=2379KiB/s (2436kB/s)(279MiB/120022msec)
slat (nsec): min=0, max=2365.6k, avg=27578.90, stdev=85435.33
clat (nsec): min=0, max=660093k, avg=10380038.22, stdev=49666588.17
lat (nsec): min=0, max=660371k, avg=10408597.54, stdev=49675165.24
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
| 20.00th=[ 756], 30.00th=[ 868], 40.00th=[ 964],
| 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1448],
| 80.00th=[ 94720], 90.00th=[ 3555328], 95.00th=[ 53739520],
| 99.00th=[278921216], 99.50th=[408944640], 99.90th=[541065216],
| 99.95th=[608174080], 99.99th=[658505728]
bw ( KiB/s): min= 0, max= 2746, per=11.31%, avg=2379.00, stdev=519.02, samples=2
iops : min= 0, max= 42, avg=36.50, stdev= 7.78, samples=2
write: IOPS=38, BW=2480KiB/s (2540kB/s)(291MiB/120022msec)
slat (nsec): min=0, max=421952k, avg=357499.62, stdev=8700755.43
clat (nsec): min=0, max=1108.1M, avg=93392464.33, stdev=148165561.77
lat (nsec): min=0, max=1108.1M, avg=93751123.75, stdev=148302960.36
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1500],
| 20.00th=[ 2180], 30.00th=[ 3884], 40.00th=[ 9765],
| 50.00th=[ 42206], 60.00th=[ 78119], 70.00th=[ 103285],
| 80.00th=[ 133694], 90.00th=[ 235930], 95.00th=[ 396362],
| 99.00th=[ 734004], 99.50th=[ 868221], 99.90th=[1061159],
| 99.95th=[1098908], 99.99th=[1115685]
bw ( KiB/s): min= 0, max= 2824, per=45.67%, avg=2478.00, stdev=489.32, samples=2
iops : min= 0, max= 44, avg=38.50, stdev= 7.78, samples=2
lat (nsec) : 750=9.30%, 1000=11.41%
lat (usec) : 2=15.74%, 4=0.14%, 10=0.26%, 20=0.79%, 50=0.70%
lat (usec) : 100=0.91%, 250=0.64%, 500=0.68%, 750=0.66%, 1000=1.39%
lat (msec) : 2=9.94%, 4=7.20%, 10=5.30%, 20=2.69%, 50=4.98%
lat (msec) : 100=9.83%, 250=12.06%, 500=3.59%, 750=1.35%, 1000=0.34%
cpu : usr=0.06%, sys=0.22%, ctx=10593, majf=0, minf=35
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4464,4648,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=20.5MiB/s (21.5MB/s), 2379KiB/s-18.2MiB/s (2436kB/s-19.1MB/s), io=2465MiB (2585MB), run=120022-120030msec
WRITE: bw=5426KiB/s (5557kB/s), 2480KiB/s-2948KiB/s (2540kB/s-3018kB/s), io=636MiB (667MB), run=120022-120088msec
Disk stats (read/write):
rbd329: ios=9355/10893, merge=0/1, ticks=557544/828062, in_queue=1375322, util=14.62%
<example-benchmark-fio-server-714-benchmark-g6ljr>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=49, BW=3166KiB/s (3242kB/s)(371MiB/120038msec)
slat (nsec): min=0, max=483547k, avg=436729.22, stdev=9798160.86
clat (nsec): min=0, max=1052.2M, avg=80745876.19, stdev=137146591.70
lat (nsec): min=0, max=1052.3M, avg=81183814.06, stdev=137524748.71
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
| 20.00th=[ 1909], 30.00th=[ 2802], 40.00th=[ 5145],
| 50.00th=[ 19268], 60.00th=[ 59507], 70.00th=[ 89654],
| 80.00th=[ 116917], 90.00th=[ 208667], 95.00th=[ 404751],
| 99.00th=[ 675283], 99.50th=[ 742392], 99.90th=[ 926942],
| 99.95th=[1027605], 99.99th=[1052771]
bw ( KiB/s): min= 0, max= 4302, per=54.75%, avg=3164.50, stdev=1608.67, samples=2
iops : min= 0, max= 67, avg=49.00, stdev=25.46, samples=2
lat (usec) : 4=0.03%, 100=0.02%, 750=0.07%, 1000=0.91%
lat (msec) : 2=20.52%, 4=14.91%, 10=9.35%, 20=4.46%, 50=7.40%
lat (msec) : 100=16.83%, 250=17.62%, 500=4.94%, 750=2.53%, 1000=0.40%
cpu : usr=0.06%, sys=0.22%, ctx=11193, majf=0, minf=29
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5936,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=300, BW=18.8MiB/s (19.7MB/s)(2254MiB/120012msec)
slat (nsec): min=0, max=484975k, avg=65282.63, stdev=4143390.88
clat (nsec): min=0, max=1203.3M, avg=13303433.72, stdev=63562701.91
lat (nsec): min=0, max=1203.6M, avg=13369542.88, stdev=63721578.58
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
| 20.00th=[ 732], 30.00th=[ 740], 40.00th=[ 756],
| 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1128],
| 80.00th=[ 23424], 90.00th=[ 13434880], 95.00th=[ 60555264],
| 99.00th=[ 367001600], 99.50th=[ 476053504], 99.90th=[ 700448768],
| 99.95th=[ 792723456], 99.99th=[1044381696]
bw ( KiB/s): min= 0, max=25968, per=88.44%, avg=19235.50, stdev=9521.19, samples=2
iops : min= 0, max= 406, avg=300.50, stdev=149.20, samples=2
lat (nsec) : 750=38.01%, 1000=28.33%
lat (usec) : 2=10.58%, 4=0.05%, 10=1.86%, 20=1.10%, 50=1.26%
lat (usec) : 100=0.10%, 250=0.07%, 500=0.09%, 750=0.86%, 1000=2.03%
lat (msec) : 2=2.74%, 4=1.86%, 10=0.90%, 20=0.53%, 50=3.54%
lat (msec) : 100=2.80%, 250=1.62%, 500=1.25%, 750=0.37%, 1000=0.05%
cpu : usr=0.09%, sys=0.35%, ctx=13433, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=36110,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=39, BW=2518KiB/s (2578kB/s)(295MiB/120029msec)
slat (nsec): min=0, max=349857k, avg=101363.58, stdev=5089683.13
clat (nsec): min=0, max=1044.2M, avg=9561601.99, stdev=48388043.08
lat (nsec): min=0, max=1044.2M, avg=9663946.77, stdev=48647228.37
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 996],
| 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
| 80.00th=[ 43776], 90.00th=[ 2899968], 95.00th=[ 51118080],
| 99.00th=[ 248512512], 99.50th=[ 379584512], 99.90th=[ 566231040],
| 99.95th=[ 666894336], 99.99th=[1044381696]
bw ( KiB/s): min= 0, max= 3013, per=11.57%, avg=2517.50, stdev=700.74, samples=2
iops : min= 0, max= 47, avg=39.00, stdev=11.31, samples=2
write: IOPS=40, BW=2614KiB/s (2677kB/s)(306MiB/120029msec)
slat (nsec): min=0, max=764714k, avg=504417.57, stdev=13821816.46
clat (nsec): min=0, max=1313.0M, avg=88688665.47, stdev=146690691.98
lat (nsec): min=0, max=1313.1M, avg=89194015.61, stdev=147209124.43
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1254], 10.00th=[ 1516],
| 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 8586],
| 50.00th=[ 33817], 60.00th=[ 67634], 70.00th=[ 93848],
| 80.00th=[ 123208], 90.00th=[ 227541], 95.00th=[ 438305],
| 99.00th=[ 683672], 99.50th=[ 775947], 99.90th=[1044382],
| 99.95th=[1061159], 99.99th=[1317012]
bw ( KiB/s): min= 0, max= 3117, per=45.20%, avg=2612.50, stdev=713.47, samples=2
iops : min= 0, max= 48, avg=40.00, stdev=11.31, samples=2
lat (nsec) : 750=8.05%, 1000=11.86%
lat (usec) : 2=17.28%, 4=0.05%, 10=0.37%, 20=1.02%, 50=0.76%
lat (usec) : 100=0.80%, 250=0.45%, 500=0.43%, 750=0.39%, 1000=1.44%
lat (msec) : 2=9.94%, 4=7.65%, 10=5.33%, 20=2.76%, 50=5.62%
lat (msec) : 100=10.70%, 250=10.22%, 500=2.99%, 750=1.54%, 1000=0.20%
cpu : usr=0.06%, sys=0.25%, ctx=11131, majf=0, minf=36
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4725,4900,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=21.2MiB/s (22.3MB/s), 2518KiB/s-18.8MiB/s (2578kB/s-19.7MB/s), io=2550MiB (2673MB), run=120012-120029msec
WRITE: bw=5781KiB/s (5919kB/s), 2614KiB/s-3166KiB/s (2677kB/s-3242kB/s), io=678MiB (711MB), run=120029-120038msec
Disk stats (read/write):
rbd292: ios=7536/10929, merge=0/2, ticks=534002/809056, in_queue=1333704, util=13.66%
<example-benchmark-fio-server-386-benchmark-nzznc>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=40, BW=2596KiB/s (2658kB/s)(304MiB/120024msec)
slat (nsec): min=0, max=579931k, avg=462348.78, stdev=11856183.51
clat (nsec): min=0, max=1090.2M, avg=98730771.42, stdev=156908561.14
lat (nsec): min=0, max=1289.8M, avg=99194171.74, stdev=157794900.50
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1418],
| 20.00th=[ 1991], 30.00th=[ 3228], 40.00th=[ 7242],
| 50.00th=[ 30802], 60.00th=[ 72877], 70.00th=[ 100140],
| 80.00th=[ 131597], 90.00th=[ 316670], 95.00th=[ 484443],
| 99.00th=[ 708838], 99.50th=[ 742392], 99.90th=[ 826278],
| 99.95th=[ 910164], 99.99th=[1082131]
bw ( KiB/s): min= 0, max= 3369, per=55.35%, avg=2593.50, stdev=1096.72, samples=2
iops : min= 0, max= 52, avg=40.00, stdev=16.97, samples=2
lat (usec) : 250=0.02%, 750=0.04%, 1000=0.84%
lat (msec) : 2=19.24%, 4=13.86%, 10=8.49%, 20=4.61%, 50=6.41%
lat (msec) : 100=16.55%, 250=17.58%, 500=7.77%, 750=4.17%, 1000=0.43%
cpu : usr=0.05%, sys=0.17%, ctx=9233, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4864,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=290, BW=18.1MiB/s (18.0MB/s)(2174MiB/120020msec)
slat (nsec): min=0, max=689525k, avg=62199.26, stdev=4622504.13
clat (nsec): min=0, max=1083.4M, avg=13794212.01, stdev=64703199.17
lat (nsec): min=0, max=1083.5M, avg=13857268.54, stdev=64873106.38
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
| 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1128],
| 80.00th=[ 8896], 90.00th=[ 15007744], 95.00th=[ 62128128],
| 99.00th=[ 387973120], 99.50th=[ 492830720], 99.90th=[ 692060160],
| 99.95th=[ 734003200], 99.99th=[1019215872]
bw ( KiB/s): min= 0, max=22016, per=90.23%, avg=18550.50, stdev=4900.96, samples=2
iops : min= 0, max= 344, avg=289.50, stdev=77.07, samples=2
lat (nsec) : 750=33.40%, 1000=32.84%
lat (usec) : 2=11.88%, 4=0.05%, 10=2.36%, 20=0.94%, 50=0.36%
lat (usec) : 100=0.11%, 250=0.05%, 500=0.18%, 750=0.86%, 1000=1.88%
lat (msec) : 2=2.49%, 4=1.47%, 10=0.79%, 20=0.64%, 50=3.51%
lat (msec) : 100=2.72%, 250=1.69%, 500=1.29%, 750=0.44%, 1000=0.03%
cpu : usr=0.09%, sys=0.32%, ctx=12530, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=34807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=31, BW=2009KiB/s (2057kB/s)(236MiB/120021msec)
slat (nsec): min=0, max=427153k, avg=136375.18, stdev=6958530.80
clat (nsec): min=0, max=702092k, avg=9087273.02, stdev=46342012.02
lat (nsec): min=0, max=702124k, avg=9224584.28, stdev=46846870.09
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 780], 30.00th=[ 868], 40.00th=[ 948],
| 50.00th=[ 1080], 60.00th=[ 1240], 70.00th=[ 1384],
| 80.00th=[ 31872], 90.00th=[ 1777664], 95.00th=[ 46399488],
| 99.00th=[261095424], 99.50th=[367001600], 99.90th=[549453824],
| 99.95th=[583008256], 99.99th=[700448768]
bw ( KiB/s): min= 0, max= 2533, per=9.77%, avg=2008.50, stdev=741.76, samples=2
iops : min= 0, max= 39, avg=31.00, stdev=11.31, samples=2
write: IOPS=32, BW=2091KiB/s (2141kB/s)(245MiB/120021msec)
slat (nsec): min=0, max=701141k, avg=429937.85, stdev=12610360.16
clat (nsec): min=0, max=1018.3M, avg=113826493.31, stdev=166482932.38
lat (nsec): min=0, max=1018.4M, avg=114257403.10, stdev=166748154.43
clat percentiles (usec):
| 1.00th=[ 1057], 5.00th=[ 1336], 10.00th=[ 1631],
| 20.00th=[ 2442], 30.00th=[ 4555], 40.00th=[ 17171],
| 50.00th=[ 56886], 60.00th=[ 86508], 70.00th=[ 109577],
| 80.00th=[ 156238], 90.00th=[ 358613], 95.00th=[ 513803],
| 99.00th=[ 725615], 99.50th=[ 759170], 99.90th=[ 809501],
| 99.95th=[1019216], 99.99th=[1019216]
bw ( KiB/s): min= 0, max= 2647, per=44.58%, avg=2089.00, stdev=789.13, samples=2
iops : min= 0, max= 41, avg=32.00, stdev=12.73, samples=2
lat (nsec) : 750=7.31%, 1000=13.84%
lat (usec) : 2=16.54%, 4=0.08%, 10=0.25%, 20=0.98%, 50=0.70%
lat (usec) : 100=1.03%, 250=0.57%, 500=0.57%, 750=0.57%, 1000=1.14%
lat (msec) : 2=8.69%, 4=6.87%, 10=4.38%, 20=2.67%, 50=4.96%
lat (msec) : 100=10.14%, 250=10.73%, 500=5.18%, 750=2.52%, 1000=0.29%
cpu : usr=0.04%, sys=0.19%, ctx=8800, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3768,3918,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=20.1MiB/s (21.1MB/s), 2009KiB/s-18.1MiB/s (2057kB/s-18.0MB/s), io=2410MiB (2527MB), run=120020-120021msec
WRITE: bw=4687KiB/s (4799kB/s), 2091KiB/s-2596KiB/s (2141kB/s-2658kB/s), io=549MiB (576MB), run=120021-120024msec
Disk stats (read/write):
rbd113: ios=7017/9006, merge=0/1, ticks=547807/860241, in_queue=1399971, util=11.72%
<example-benchmark-fio-server-616-benchmark-vm6nt>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=42, BW=2740KiB/s (2806kB/s)(321MiB/120064msec)
slat (nsec): min=0, max=522799k, avg=448436.58, stdev=10743519.39
clat (nsec): min=0, max=1524.9M, avg=93344224.78, stdev=161360206.70
lat (nsec): min=0, max=1524.9M, avg=93793721.83, stdev=161971351.87
clat percentiles (usec):
| 1.00th=[ 938], 5.00th=[ 1156], 10.00th=[ 1369],
| 20.00th=[ 1876], 30.00th=[ 2868], 40.00th=[ 5145],
| 50.00th=[ 19268], 60.00th=[ 60031], 70.00th=[ 87557],
| 80.00th=[ 125305], 90.00th=[ 304088], 95.00th=[ 459277],
| 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1149240],
| 99.95th=[1233126], 99.99th=[1518339]
bw ( KiB/s): min= 0, max= 1793, per=34.93%, avg=1793.00, stdev= 0.00, samples=1
iops : min= 0, max= 28, avg=28.00, stdev= 0.00, samples=1
lat (usec) : 500=0.02%, 750=0.04%, 1000=1.87%
lat (msec) : 2=19.66%, 4=14.99%, 10=9.93%, 20=3.89%, 50=6.64%
lat (msec) : 100=17.05%, 250=14.05%, 500=8.00%, 750=2.76%, 1000=0.91%
cpu : usr=0.04%, sys=0.20%, ctx=9807, majf=0, minf=32
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5138,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=383, BW=23.9MiB/s (25.1MB/s)(2872MiB/120013msec)
slat (nsec): min=0, max=471122k, avg=46847.59, stdev=3381194.72
clat (nsec): min=0, max=1407.2M, avg=10409463.13, stdev=55502628.48
lat (nsec): min=0, max=1407.2M, avg=10457142.71, stdev=55610776.23
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 972],
| 80.00th=[ 1432], 90.00th=[ 3424256], 95.00th=[ 50593792],
| 99.00th=[ 291504128], 99.50th=[ 450887680], 99.90th=[ 675282944],
| 99.95th=[ 767557632], 99.99th=[1098907648]
bw ( KiB/s): min= 0, max=34035, per=100.00%, avg=34035.00, stdev= 0.00, samples=1
iops : min= 0, max= 532, avg=532.00, stdev= 0.00, samples=1
lat (nsec) : 750=43.64%, 1000=27.38%
lat (usec) : 2=10.12%, 4=0.03%, 10=0.80%, 20=0.30%, 50=0.14%
lat (usec) : 100=0.06%, 250=0.06%, 500=0.22%, 750=0.88%, 1000=2.09%
lat (msec) : 2=2.76%, 4=1.76%, 10=0.97%, 20=0.57%, 50=3.17%
lat (msec) : 100=2.66%, 250=1.23%, 500=0.83%, 750=0.29%, 1000=0.04%
cpu : usr=0.13%, sys=0.41%, ctx=16042, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=45981,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=35, BW=2297KiB/s (2352kB/s)(269MiB/120051msec)
slat (nsec): min=0, max=3687.3k, avg=25139.08, stdev=91477.78
clat (nsec): min=0, max=1038.4M, avg=7695728.63, stdev=45093642.98
lat (nsec): min=0, max=1038.6M, avg=7721793.49, stdev=45105435.26
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
| 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1384],
| 80.00th=[ 15808], 90.00th=[ 1777664], 95.00th=[ 41156608],
| 99.00th=[ 183500800], 99.50th=[ 304087040], 99.90th=[ 666894336],
| 99.95th=[ 725614592], 99.99th=[1035993088]
bw ( KiB/s): min= 0, max= 2781, per=8.57%, avg=2297.00, stdev=684.48, samples=2
iops : min= 0, max= 43, avg=35.50, stdev=10.61, samples=2
write: IOPS=37, BW=2393KiB/s (2451kB/s)(281MiB/120051msec)
slat (nsec): min=0, max=521733k, avg=510207.12, stdev=12528343.03
clat (nsec): min=0, max=1456.0M, avg=99230619.88, stdev=167653400.45
lat (nsec): min=0, max=1456.1M, avg=99741774.10, stdev=168293190.74
clat percentiles (usec):
| 1.00th=[ 971], 5.00th=[ 1254], 10.00th=[ 1500],
| 20.00th=[ 2089], 30.00th=[ 3326], 40.00th=[ 6456],
| 50.00th=[ 23987], 60.00th=[ 61080], 70.00th=[ 88605],
| 80.00th=[ 131597], 90.00th=[ 341836], 95.00th=[ 476054],
| 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1216349],
| 99.95th=[1367344], 99.99th=[1451230]
bw ( KiB/s): min= 0, max= 2873, per=46.60%, avg=2392.00, stdev=680.24, samples=2
iops : min= 0, max= 44, avg=36.50, stdev=10.61, samples=2
lat (nsec) : 750=9.00%, 1000=12.22%
lat (usec) : 2=17.29%, 4=0.08%, 10=0.19%, 20=0.72%, 50=0.70%
lat (usec) : 100=0.61%, 250=0.48%, 500=0.49%, 750=0.44%, 1000=1.53%
lat (msec) : 2=10.24%, 4=7.73%, 10=6.29%, 20=2.58%, 50=5.10%
lat (msec) : 100=9.71%, 250=7.68%, 500=4.64%, 750=1.66%, 1000=0.50%
cpu : usr=0.05%, sys=0.21%, ctx=10071, majf=0, minf=38
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4311,4486,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=26.2MiB/s (27.4MB/s), 2297KiB/s-23.9MiB/s (2352kB/s-25.1MB/s), io=3141MiB (3294MB), run=120013-120051msec
WRITE: bw=5133KiB/s (5256kB/s), 2393KiB/s-2740KiB/s (2451kB/s-2806kB/s), io=602MiB (631MB), run=120051-120064msec
Disk stats (read/write):
rbd226: ios=8971/10050, merge=0/2, ticks=526494/837841, in_queue=1354664, util=14.05%
<example-benchmark-fio-server-810-benchmark-s6m2k>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=45, BW=2945KiB/s (3016kB/s)(345MiB/120041msec)
slat (nsec): min=0, max=573788k, avg=367710.09, stdev=9878806.33
clat (nsec): min=0, max=1214.2M, avg=86620290.25, stdev=149814104.02
lat (nsec): min=0, max=1214.3M, avg=86989216.49, stdev=149987111.38
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1418],
| 20.00th=[ 1909], 30.00th=[ 2900], 40.00th=[ 5211],
| 50.00th=[ 13435], 60.00th=[ 47973], 70.00th=[ 85459],
| 80.00th=[ 120062], 90.00th=[ 283116], 95.00th=[ 438305],
| 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 952108],
| 99.95th=[1044382], 99.99th=[1216349]
bw ( KiB/s): min= 0, max= 3560, per=55.22%, avg=2944.00, stdev=871.16, samples=2
iops : min= 0, max= 55, avg=45.50, stdev=13.44, samples=2
lat (usec) : 4=0.02%, 750=0.04%, 1000=1.00%
lat (msec) : 2=20.12%, 4=14.45%, 10=12.19%, 20=5.11%, 50=7.63%
lat (msec) : 100=14.69%, 250=13.80%, 500=7.66%, 750=2.54%, 1000=0.76%
cpu : usr=0.06%, sys=0.19%, ctx=10456, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5521,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=294, BW=18.4MiB/s (19.3MB/s)(2204MiB/120030msec)
slat (nsec): min=0, max=310412k, avg=40472.37, stdev=2303449.06
clat (nsec): min=0, max=1217.2M, avg=13618848.09, stdev=62825076.11
lat (nsec): min=0, max=1217.3M, avg=13660203.59, stdev=62875748.57
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
| 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
| 80.00th=[ 561152], 90.00th=[ 23461888], 95.00th=[ 63176704],
| 99.00th=[ 354418688], 99.50th=[ 476053504], 99.90th=[ 700448768],
| 99.95th=[ 759169024], 99.99th=[1061158912]
bw ( KiB/s): min= 0, max=24758, per=89.20%, avg=18807.00, stdev=8415.98, samples=2
iops : min= 0, max= 387, avg=293.50, stdev=132.23, samples=2
lat (nsec) : 750=35.79%, 1000=31.21%
lat (usec) : 2=11.36%, 4=0.04%, 10=0.62%, 20=0.46%, 50=0.17%
lat (usec) : 100=0.10%, 250=0.05%, 500=0.15%, 750=1.01%, 1000=2.07%
lat (msec) : 2=3.01%, 4=1.86%, 10=0.96%, 20=0.78%, 50=3.94%
lat (msec) : 100=3.04%, 250=1.73%, 500=1.21%, 750=0.38%, 1000=0.05%
cpu : usr=0.09%, sys=0.34%, ctx=14244, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=35289,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=35, BW=2281KiB/s (2336kB/s)(267MiB/120036msec)
slat (nsec): min=0, max=378161k, avg=188024.60, stdev=7512404.09
clat (nsec): min=0, max=739179k, avg=10245570.83, stdev=48682354.00
lat (nsec): min=0, max=739211k, avg=10434521.92, stdev=49233760.49
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 772], 30.00th=[ 868], 40.00th=[ 980],
| 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1416],
| 80.00th=[ 56576], 90.00th=[ 4014080], 95.00th=[ 52690944],
| 99.00th=[283115520], 99.50th=[354418688], 99.90th=[557842432],
| 99.95th=[616562688], 99.99th=[742391808]
bw ( KiB/s): min= 0, max= 2588, per=10.82%, avg=2281.50, stdev=433.46, samples=2
iops : min= 0, max= 40, avg=35.00, stdev= 7.07, samples=2
write: IOPS=37, BW=2386KiB/s (2444kB/s)(280MiB/120036msec)
slat (nsec): min=0, max=6032.8k, avg=173025.17, stdev=261709.00
clat (nsec): min=0, max=1270.0M, avg=97171527.70, stdev=157884329.78
lat (nsec): min=0, max=1270.3M, avg=97346023.25, stdev=157836529.25
clat percentiles (usec):
| 1.00th=[ 1029], 5.00th=[ 1270], 10.00th=[ 1532],
| 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 7701],
| 50.00th=[ 24773], 60.00th=[ 64750], 70.00th=[ 94897],
| 80.00th=[ 135267], 90.00th=[ 312476], 95.00th=[ 476054],
| 99.00th=[ 742392], 99.50th=[ 843056], 99.90th=[ 935330],
| 99.95th=[ 952108], 99.99th=[1266680]
bw ( KiB/s): min= 0, max= 2692, per=44.73%, avg=2384.50, stdev=434.87, samples=2
iops : min= 0, max= 42, avg=37.00, stdev= 7.07, samples=2
lat (nsec) : 750=8.44%, 1000=11.81%
lat (usec) : 2=17.02%, 4=0.05%, 10=0.14%, 20=0.80%, 50=0.69%
lat (usec) : 100=0.74%, 250=0.49%, 500=0.41%, 750=0.56%, 1000=1.36%
lat (msec) : 2=9.93%, 4=7.71%, 10=5.94%, 20=3.15%, 50=5.67%
lat (msec) : 100=9.30%, 250=8.91%, 500=4.67%, 750=1.75%, 1000=0.47%
cpu : usr=0.06%, sys=0.21%, ctx=10250, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4280,4473,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=20.6MiB/s (21.6MB/s), 2281KiB/s-18.4MiB/s (2336kB/s-19.3MB/s), io=2472MiB (2592MB), run=120030-120036msec
WRITE: bw=5332KiB/s (5459kB/s), 2386KiB/s-2945KiB/s (2444kB/s-3016kB/s), io=625MiB (655MB), run=120036-120041msec
Disk stats (read/write):
rbd188: ios=7908/10343, merge=0/1, ticks=544904/835140, in_queue=1370788, util=13.49%
<example-benchmark-fio-server-363-benchmark-v4496>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=41, BW=2646KiB/s (2709kB/s)(310MiB/120098msec)
slat (nsec): min=0, max=483563k, avg=335624.44, stdev=7666597.67
clat (nsec): min=0, max=1102.5M, avg=96744121.92, stdev=159436843.91
lat (nsec): min=0, max=1102.5M, avg=97080769.02, stdev=159472833.05
clat percentiles (usec):
| 1.00th=[ 938], 5.00th=[ 1106], 10.00th=[ 1303],
| 20.00th=[ 1663], 30.00th=[ 2343], 40.00th=[ 3818],
| 50.00th=[ 23200], 60.00th=[ 72877], 70.00th=[ 103285],
| 80.00th=[ 129500], 90.00th=[ 308282], 95.00th=[ 488637],
| 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[ 901776],
| 99.95th=[ 935330], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 4098, per=55.36%, avg=2645.50, stdev=2054.15, samples=2
iops : min= 0, max= 64, avg=41.00, stdev=32.53, samples=2
lat (usec) : 2=0.02%, 4=0.04%, 500=0.02%, 750=0.08%, 1000=1.67%
lat (msec) : 2=24.00%, 4=14.89%, 10=6.15%, 20=2.56%, 50=4.90%
lat (msec) : 100=14.53%, 250=19.87%, 500=6.59%, 750=3.89%, 1000=0.81%
cpu : usr=0.05%, sys=0.17%, ctx=9368, majf=0, minf=27
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4962,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=323, BW=20.2MiB/s (21.2MB/s)(2424MiB/120013msec)
slat (nsec): min=0, max=477001k, avg=36573.14, stdev=2637904.02
clat (nsec): min=0, max=940724k, avg=12367744.96, stdev=60911558.88
lat (nsec): min=0, max=941214k, avg=12405146.29, stdev=60976937.14
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 796], 60.00th=[ 860], 70.00th=[ 940],
| 80.00th=[ 1304], 90.00th=[ 6127616], 95.00th=[ 55836672],
| 99.00th=[367001600], 99.50th=[471859200], 99.90th=[700448768],
| 99.95th=[767557632], 99.99th=[843055104]
bw ( KiB/s): min= 0, max=25084, per=91.05%, avg=20685.00, stdev=6221.13, samples=2
iops : min= 0, max= 392, avg=323.00, stdev=97.58, samples=2
lat (nsec) : 750=44.89%, 1000=27.73%
lat (usec) : 2=9.91%, 4=0.08%, 10=0.35%, 20=0.21%, 50=0.19%
lat (usec) : 100=0.06%, 250=0.04%, 500=0.16%, 750=0.79%, 1000=1.59%
lat (msec) : 2=2.21%, 4=1.37%, 10=0.80%, 20=0.56%, 50=3.41%
lat (msec) : 100=2.67%, 250=1.38%, 500=1.21%, 750=0.34%, 1000=0.06%
cpu : usr=0.10%, sys=0.35%, ctx=12840, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=38815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=31, BW=2033KiB/s (2082kB/s)(238MiB/120015msec)
slat (nsec): min=0, max=436760k, avg=141784.50, stdev=7073229.93
clat (nsec): min=0, max=698805k, avg=9375440.61, stdev=44524086.18
lat (nsec): min=0, max=698827k, avg=9518114.96, stdev=45063482.36
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
| 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 988],
| 50.00th=[ 1112], 60.00th=[ 1224], 70.00th=[ 1416],
| 80.00th=[ 67072], 90.00th=[ 5537792], 95.00th=[ 54788096],
| 99.00th=[221249536], 99.50th=[325058560], 99.90th=[599785472],
| 99.95th=[683671552], 99.99th=[700448768]
bw ( KiB/s): min= 0, max= 3070, per=8.94%, avg=2032.00, stdev=1467.95, samples=2
iops : min= 0, max= 47, avg=31.00, stdev=22.63, samples=2
write: IOPS=33, BW=2135KiB/s (2186kB/s)(250MiB/120015msec)
slat (nsec): min=0, max=192477k, avg=220163.16, stdev=3048609.65
clat (nsec): min=0, max=1102.4M, avg=110902683.24, stdev=169976312.05
lat (nsec): min=0, max=1102.6M, avg=111123871.68, stdev=169947686.95
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1369],
| 20.00th=[ 1926], 30.00th=[ 3032], 40.00th=[ 8586],
| 50.00th=[ 54264], 60.00th=[ 86508], 70.00th=[ 111674],
| 80.00th=[ 141558], 90.00th=[ 350225], 95.00th=[ 522191],
| 99.00th=[ 750781], 99.50th=[ 843056], 99.90th=[ 926942],
| 99.95th=[ 935330], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 3198, per=44.64%, avg=2133.50, stdev=1505.43, samples=2
iops : min= 0, max= 49, avg=32.50, stdev=23.33, samples=2
lat (nsec) : 750=5.14%, 1000=14.97%
lat (usec) : 2=16.56%, 4=0.08%, 10=0.14%, 20=0.82%, 50=0.78%
lat (usec) : 100=1.10%, 250=0.52%, 500=0.59%, 750=0.64%, 1000=1.61%
lat (msec) : 2=11.03%, 4=7.50%, 10=3.43%, 20=1.87%, 50=4.54%
lat (msec) : 100=9.75%, 250=11.70%, 500=4.33%, 750=2.42%, 1000=0.50%
cpu : usr=0.05%, sys=0.19%, ctx=9130, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3814,4000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=22.2MiB/s (23.3MB/s), 2033KiB/s-20.2MiB/s (2082kB/s-21.2MB/s), io=2663MiB (2792MB), run=120013-120015msec
WRITE: bw=4779KiB/s (4894kB/s), 2135KiB/s-2646KiB/s (2186kB/s-2709kB/s), io=561MiB (588MB), run=120015-120098msec
Disk stats (read/write):
rbd141: ios=7942/9637, merge=0/3, ticks=548741/850068, in_queue=1389903, util=12.74%
Jobs: 63 (f=59): [_(2),m(1)][99.8%][r=1231M,w=976M][r=19.2k,w=15.2k IOPS][eta 00m:55s] <example-benchmark-fio-server-910-benchmark-74gxv>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=47, BW=3049KiB/s (3122kB/s)(358MiB/120101msec)
slat (nsec): min=0, max=587401k, avg=499377.44, stdev=12120076.80
clat (nsec): min=0, max=1699.1M, avg=83679046.09, stdev=152339670.45
lat (nsec): min=0, max=1699.1M, avg=84179546.98, stdev=152850121.23
clat percentiles (usec):
| 1.00th=[ 979], 5.00th=[ 1156], 10.00th=[ 1336],
| 20.00th=[ 1778], 30.00th=[ 2638], 40.00th=[ 4686],
| 50.00th=[ 14222], 60.00th=[ 51643], 70.00th=[ 87557],
| 80.00th=[ 117965], 90.00th=[ 225444], 95.00th=[ 413139],
| 99.00th=[ 725615], 99.50th=[ 884999], 99.90th=[1149240],
| 99.95th=[1300235], 99.99th=[1702888]
bw ( KiB/s): min= 0, max= 3874, per=54.17%, avg=3049.50, stdev=1166.02, samples=2
iops : min= 0, max= 60, avg=47.00, stdev=18.38, samples=2
lat (usec) : 750=0.07%, 1000=1.42%
lat (msec) : 2=21.72%, 4=14.44%, 10=10.21%, 20=4.77%, 50=6.92%
lat (msec) : 100=14.43%, 250=16.89%, 500=5.58%, 750=2.64%, 1000=0.65%
cpu : usr=0.06%, sys=0.20%, ctx=10898, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5719,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=369, BW=23.0MiB/s (24.2MB/s)(2766MiB/120003msec)
slat (nsec): min=0, max=586968k, avg=49643.35, stdev=3526447.54
clat (nsec): min=0, max=1687.4M, avg=10816930.67, stdev=56864630.85
lat (nsec): min=0, max=1687.5M, avg=10867413.28, stdev=56980443.13
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 812], 60.00th=[ 868], 70.00th=[ 948],
| 80.00th=[ 1384], 90.00th=[ 3555328], 95.00th=[ 50593792],
| 99.00th=[ 308281344], 99.50th=[ 438304768], 99.90th=[ 658505728],
| 99.95th=[ 734003200], 99.99th=[1233125376]
bw ( KiB/s): min= 0, max=34327, per=90.47%, avg=23600.50, stdev=15169.56, samples=2
iops : min= 0, max= 537, avg=369.00, stdev=237.59, samples=2
lat (nsec) : 750=43.42%, 1000=28.81%
lat (usec) : 2=9.35%, 4=0.04%, 10=0.87%, 20=1.08%, 50=0.38%
lat (usec) : 100=0.12%, 250=0.07%, 500=0.10%, 750=0.73%, 1000=1.59%
lat (msec) : 2=2.31%, 4=1.29%, 10=0.76%, 20=0.71%, 50=3.29%
lat (msec) : 100=2.40%, 250=1.39%, 500=0.95%, 750=0.28%, 1000=0.03%
cpu : usr=0.10%, sys=0.41%, ctx=14046, majf=0, minf=97
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=44330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=38, BW=2487KiB/s (2547kB/s)(292MiB/120019msec)
slat (nsec): min=0, max=368100k, avg=106961.37, stdev=5389940.62
clat (nsec): min=0, max=759558k, avg=9002016.33, stdev=40516136.77
lat (nsec): min=0, max=759588k, avg=9109980.22, stdev=40868878.36
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 996],
| 50.00th=[ 1144], 60.00th=[ 1256], 70.00th=[ 1432],
| 80.00th=[ 71168], 90.00th=[ 5275648], 95.00th=[ 54263808],
| 99.00th=[212860928], 99.50th=[308281344], 99.90th=[429916160],
| 99.95th=[476053504], 99.99th=[759169024]
bw ( KiB/s): min= 0, max= 2916, per=9.53%, avg=2486.50, stdev=607.40, samples=2
iops : min= 0, max= 45, avg=38.50, stdev= 9.19, samples=2
write: IOPS=40, BW=2583KiB/s (2645kB/s)(303MiB/120019msec)
slat (nsec): min=0, max=626891k, avg=490839.66, stdev=12588951.98
clat (nsec): min=0, max=1796.9M, avg=90155001.53, stdev=160192506.95
lat (nsec): min=0, max=1796.0M, avg=90646821.59, stdev=160496253.46
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
| 20.00th=[ 2040], 30.00th=[ 3294], 40.00th=[ 6587],
| 50.00th=[ 21627], 60.00th=[ 59507], 70.00th=[ 91751],
| 80.00th=[ 121111], 90.00th=[ 252707], 95.00th=[ 442500],
| 99.00th=[ 759170], 99.50th=[ 884999], 99.90th=[1300235],
| 99.95th=[1686111], 99.99th=[1803551]
bw ( KiB/s): min= 0, max= 3015, per=45.85%, avg=2581.50, stdev=613.06, samples=2
iops : min= 0, max= 47, avg=40.00, stdev= 9.90, samples=2
lat (nsec) : 750=7.48%, 1000=12.19%
lat (usec) : 2=17.28%, 4=0.13%, 10=0.34%, 20=0.79%, 50=0.62%
lat (usec) : 100=0.64%, 250=0.64%, 500=0.63%, 750=0.41%, 1000=1.38%
lat (msec) : 2=10.86%, 4=7.39%, 10=6.11%, 20=3.06%, 50=5.52%
lat (msec) : 100=9.36%, 250=9.71%, 500=3.46%, 750=1.48%, 1000=0.39%
cpu : usr=0.05%, sys=0.24%, ctx=11003, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4666,4841,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=25.5MiB/s (26.7MB/s), 2487KiB/s-23.0MiB/s (2547kB/s-24.2MB/s), io=3058MiB (3206MB), run=120003-120019msec
WRITE: bw=5630KiB/s (5766kB/s), 2583KiB/s-3049KiB/s (2645kB/s-3122kB/s), io=660MiB (692MB), run=120019-120101msec
Disk stats (read/write):
rbd320: ios=8138/10744, merge=0/1, ticks=543117/803685, in_queue=1337178, util=13.96%
<example-benchmark-fio-server-697-benchmark-v5m2j>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=52, BW=3334KiB/s (3414kB/s)(391MiB/120024msec)
slat (nsec): min=0, max=950382k, avg=452969.44, stdev=14552434.79
clat (nsec): min=0, max=1440.0M, avg=76854293.56, stdev=153490325.63
lat (nsec): min=0, max=1441.2M, avg=77308421.84, stdev=154089812.05
clat percentiles (usec):
| 1.00th=[ 955], 5.00th=[ 1205], 10.00th=[ 1401],
| 20.00th=[ 1762], 30.00th=[ 2376], 40.00th=[ 3589],
| 50.00th=[ 6849], 60.00th=[ 23725], 70.00th=[ 76022],
| 80.00th=[ 107480], 90.00th=[ 208667], 95.00th=[ 392168],
| 99.00th=[ 784335], 99.50th=[ 910164], 99.90th=[1199571],
| 99.95th=[1300235], 99.99th=[1434452]
bw ( KiB/s): min= 0, max= 5185, per=58.75%, avg=3333.00, stdev=2619.12, samples=2
iops : min= 0, max= 81, avg=52.00, stdev=41.01, samples=2
lat (usec) : 100=0.02%, 250=0.03%, 750=0.03%, 1000=1.36%
lat (msec) : 2=23.31%, 4=17.55%, 10=11.94%, 20=4.78%, 50=4.61%
lat (msec) : 100=14.32%, 250=13.89%, 500=4.88%, 750=2.08%, 1000=0.99%
cpu : usr=0.06%, sys=0.22%, ctx=11738, majf=0, minf=27
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6250,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=341, BW=21.3MiB/s (22.4MB/s)(2561MiB/120020msec)
slat (nsec): min=0, max=871267k, avg=58333.28, stdev=5300696.13
clat (nsec): min=0, max=1267.3M, avg=11691015.06, stdev=59871844.84
lat (nsec): min=0, max=1267.5M, avg=11750191.55, stdev=60112618.15
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
| 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1032],
| 80.00th=[ 1576], 90.00th=[ 4882432], 95.00th=[ 53739520],
| 99.00th=[ 337641472], 99.50th=[ 467664896], 99.90th=[ 692060160],
| 99.95th=[ 784334848], 99.99th=[1082130432]
bw ( KiB/s): min= 0, max=28268, per=90.74%, avg=21854.00, stdev=9070.77, samples=2
iops : min= 0, max= 442, avg=341.50, stdev=142.13, samples=2
lat (nsec) : 750=36.43%, 1000=32.41%
lat (usec) : 2=11.69%, 4=0.06%, 10=0.56%, 20=0.22%, 50=0.16%
lat (usec) : 100=0.09%, 250=0.04%, 500=0.25%, 750=1.03%, 1000=2.28%
lat (msec) : 2=2.99%, 4=1.55%, 10=0.76%, 20=0.64%, 50=3.41%
lat (msec) : 100=2.65%, 250=1.38%, 500=1.02%, 750=0.31%, 1000=0.04%
cpu : usr=0.11%, sys=0.39%, ctx=14938, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41006,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=34, BW=2232KiB/s (2286kB/s)(262MiB/120028msec)
slat (nsec): min=0, max=3261.9k, avg=30353.32, stdev=99290.01
clat (nsec): min=0, max=1235.7M, avg=9591222.87, stdev=49457877.63
lat (nsec): min=0, max=1235.9M, avg=9622550.11, stdev=49466608.08
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 1020],
| 50.00th=[ 1160], 60.00th=[ 1288], 70.00th=[ 1496],
| 80.00th=[ 193536], 90.00th=[ 6127616], 95.00th=[ 52690944],
| 99.00th=[ 235929600], 99.50th=[ 354418688], 99.90th=[ 517996544],
| 99.95th=[ 641728512], 99.99th=[1233125376]
bw ( KiB/s): min= 0, max= 3177, per=9.27%, avg=2232.00, stdev=1336.43, samples=2
iops : min= 0, max= 49, avg=34.50, stdev=20.51, samples=2
write: IOPS=36, BW=2340KiB/s (2396kB/s)(274MiB/120028msec)
slat (nsec): min=0, max=839359k, avg=446443.00, stdev=13829654.09
clat (nsec): min=0, max=1464.5M, avg=100143464.64, stdev=174579671.22
lat (nsec): min=0, max=1464.6M, avg=100590859.57, stdev=175258011.37
clat percentiles (usec):
| 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1532],
| 20.00th=[ 2212], 30.00th=[ 3589], 40.00th=[ 7373],
| 50.00th=[ 29754], 60.00th=[ 71828], 70.00th=[ 95945],
| 80.00th=[ 125305], 90.00th=[ 278922], 95.00th=[ 484443],
| 99.00th=[ 884999], 99.50th=[ 943719], 99.90th=[1350566],
| 99.95th=[1468007], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 3323, per=41.20%, avg=2337.50, stdev=1393.71, samples=2
iops : min= 0, max= 51, avg=36.00, stdev=21.21, samples=2
lat (nsec) : 750=7.20%, 1000=11.69%
lat (usec) : 2=17.30%, 4=0.08%, 10=0.29%, 20=0.59%, 50=0.77%
lat (usec) : 100=0.80%, 250=0.59%, 500=0.42%, 750=0.65%, 1000=1.42%
lat (msec) : 2=10.28%, 4=8.01%, 10=6.04%, 20=2.99%, 50=4.37%
lat (msec) : 100=11.07%, 250=9.46%, 500=3.51%, 750=1.54%, 1000=0.75%
cpu : usr=0.06%, sys=0.21%, ctx=10176, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4187,4385,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.5MiB/s (24.7MB/s), 2232KiB/s-21.3MiB/s (2286kB/s-22.4MB/s), io=2823MiB (2960MB), run=120020-120028msec
WRITE: bw=5674KiB/s (5810kB/s), 2340KiB/s-3334KiB/s (2396kB/s-3414kB/s), io=665MiB (697MB), run=120024-120028msec
Disk stats (read/write):
rbd234: ios=8956/11204, merge=0/1, ticks=542224/827259, in_queue=1359118, util=14.50%
<example-benchmark-fio-server-112-benchmark-wpgw6>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=47, BW=3020KiB/s (3092kB/s)(354MiB/120030msec)
slat (nsec): min=0, max=434516k, avg=325543.33, stdev=7343542.39
clat (nsec): min=0, max=953098k, avg=84734749.94, stdev=150303929.00
lat (nsec): min=0, max=953156k, avg=85061546.03, stdev=150791631.58
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401], 20.00th=[ 1844],
| 30.00th=[ 2573], 40.00th=[ 4113], 50.00th=[ 8848], 60.00th=[ 36963],
| 70.00th=[ 79168], 80.00th=[119014], 90.00th=[295699], 95.00th=[459277],
| 99.00th=[683672], 99.50th=[725615], 99.90th=[792724], 99.95th=[876610],
| 99.99th=[952108]
bw ( KiB/s): min= 0, max= 4542, per=57.71%, avg=3017.50, stdev=2155.97, samples=2
iops : min= 0, max= 70, avg=46.50, stdev=33.23, samples=2
lat (usec) : 4=0.02%, 50=0.02%, 250=0.02%, 500=0.04%, 750=0.02%
lat (usec) : 1000=0.97%
lat (msec) : 2=21.73%, 4=16.86%, 10=11.47%, 20=4.84%, 50=6.17%
lat (msec) : 100=13.85%, 250=13.06%, 500=7.26%, 750=3.43%, 1000=0.32%
cpu : usr=0.05%, sys=0.19%, ctx=10421, majf=0, minf=25
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5660,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2523MiB/120021msec)
slat (nsec): min=0, max=406062k, avg=39732.49, stdev=2589915.75
clat (nsec): min=0, max=1114.5M, avg=11880232.54, stdev=59995718.21
lat (nsec): min=0, max=1114.6M, avg=11920800.50, stdev=60062257.19
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 796], 60.00th=[ 868], 70.00th=[ 972],
| 80.00th=[ 1432], 90.00th=[ 3620864], 95.00th=[ 53739520],
| 99.00th=[362807296], 99.50th=[471859200], 99.90th=[675282944],
| 99.95th=[742391808], 99.99th=[901775360]
bw ( KiB/s): min= 0, max=27225, per=91.10%, avg=21528.00, stdev=8056.77, samples=2
iops : min= 0, max= 425, avg=336.00, stdev=125.87, samples=2
lat (nsec) : 750=44.70%, 1000=26.30%
lat (usec) : 2=10.01%, 4=0.04%, 10=0.50%, 20=0.32%, 50=0.16%
lat (usec) : 100=0.06%, 250=0.06%, 500=0.29%, 750=1.02%, 1000=2.18%
lat (msec) : 2=2.88%, 4=1.61%, 10=0.76%, 20=0.62%, 50=3.14%
lat (msec) : 100=2.47%, 250=1.40%, 500=1.09%, 750=0.37%, 1000=0.04%
cpu : usr=0.11%, sys=0.37%, ctx=14426, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=32, BW=2106KiB/s (2156kB/s)(247MiB/120003msec)
slat (nsec): min=0, max=3284.3k, avg=29352.61, stdev=98817.29
clat (nsec): min=0, max=785351k, avg=10373089.41, stdev=48731284.43
lat (nsec): min=0, max=785379k, avg=10403500.78, stdev=48741066.99
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 772], 30.00th=[ 884], 40.00th=[ 1004],
| 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1480],
| 80.00th=[ 154624], 90.00th=[ 11337728], 95.00th=[ 54788096],
| 99.00th=[265289728], 99.50th=[387973120], 99.90th=[591396864],
| 99.95th=[616562688], 99.99th=[784334848]
bw ( KiB/s): min= 0, max= 3043, per=8.91%, avg=2104.50, stdev=1327.24, samples=2
iops : min= 0, max= 47, avg=32.50, stdev=20.51, samples=2
write: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120003msec)
slat (nsec): min=0, max=392423k, avg=356288.72, stdev=8053940.39
clat (nsec): min=0, max=1000.3M, avg=105913790.38, stdev=167085986.59
lat (nsec): min=0, max=1000.4M, avg=106271139.65, stdev=167201360.65
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1467],
| 20.00th=[ 2114], 30.00th=[ 3392], 40.00th=[ 6980],
| 50.00th=[ 27395], 60.00th=[ 68682], 70.00th=[ 99091],
| 80.00th=[ 147850], 90.00th=[ 371196], 95.00th=[ 497026],
| 99.00th=[ 717226], 99.50th=[ 750781], 99.90th=[ 876610],
| 99.95th=[ 876610], 99.99th=[1002439]
bw ( KiB/s): min= 0, max= 3193, per=42.24%, avg=2208.50, stdev=1392.29, samples=2
iops : min= 0, max= 49, avg=34.00, stdev=21.21, samples=2
lat (nsec) : 750=8.26%, 1000=11.10%
lat (usec) : 2=16.60%, 4=0.10%, 10=0.26%, 20=0.94%, 50=0.78%
lat (usec) : 100=0.75%, 250=0.61%, 500=0.54%, 750=0.48%, 1000=1.27%
lat (msec) : 2=10.62%, 4=7.74%, 10=6.02%, 20=2.82%, 50=5.01%
lat (msec) : 100=9.83%, 250=8.37%, 500=5.39%, 750=2.29%, 1000=0.27%
cpu : usr=0.05%, sys=0.19%, ctx=9532, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3949,4142,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.1MiB/s (24.2MB/s), 2106KiB/s-21.0MiB/s (2156kB/s-22.0MB/s), io=2770MiB (2904MB), run=120003-120021msec
WRITE: bw=5230KiB/s (5355kB/s), 2211KiB/s-3020KiB/s (2264kB/s-3092kB/s), io=613MiB (643MB), run=120003-120030msec
Disk stats (read/write):
rbd37: ios=8286/10343, merge=0/3, ticks=542121/837826, in_queue=1370444, util=13.14%
<example-benchmark-fio-server-5-benchmark-hchc9>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=43, BW=2784KiB/s (2851kB/s)(326MiB/120043msec)
slat (nsec): min=0, max=1469.3M, avg=542823.26, stdev=21352335.09
clat (nsec): min=0, max=1719.5M, avg=91545953.17, stdev=150529149.13
lat (nsec): min=0, max=2232.0M, avg=92089805.76, stdev=153224563.06
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1237], 10.00th=[ 1418],
| 20.00th=[ 1958], 30.00th=[ 3228], 40.00th=[ 6980],
| 50.00th=[ 42730], 60.00th=[ 78119], 70.00th=[ 102237],
| 80.00th=[ 127402], 90.00th=[ 219153], 95.00th=[ 408945],
| 99.00th=[ 725615], 99.50th=[ 834667], 99.90th=[1350566],
| 99.95th=[1468007], 99.99th=[1719665]
bw ( KiB/s): min= 0, max= 3203, per=54.41%, avg=2782.50, stdev=594.68, samples=2
iops : min= 0, max= 50, avg=43.00, stdev= 9.90, samples=2
lat (usec) : 4=0.02%, 750=0.11%, 1000=1.03%
lat (msec) : 2=19.37%, 4=12.93%, 10=9.37%, 20=3.37%, 50=5.31%
lat (msec) : 100=17.24%, 250=22.05%, 500=6.05%, 750=2.28%, 1000=0.73%
cpu : usr=0.05%, sys=0.18%, ctx=9915, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5219,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=348, BW=21.8MiB/s (22.9MB/s)(2615MiB/120001msec)
slat (nsec): min=0, max=1466.4M, avg=67849.19, stdev=7423963.40
clat (nsec): min=0, max=1973.0M, avg=11410932.15, stdev=57537264.39
lat (nsec): min=0, max=1974.0M, avg=11479646.23, stdev=58075993.63
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 780],
| 50.00th=[ 852], 60.00th=[ 932], 70.00th=[ 1192],
| 80.00th=[ 839680], 90.00th=[ 14352384], 95.00th=[ 54788096],
| 99.00th=[ 304087040], 99.50th=[ 438304768], 99.90th=[ 683671552],
| 99.95th=[ 792723456], 99.99th=[1216348160]
bw ( KiB/s): min= 0, max=31764, per=90.94%, avg=22314.00, stdev=13364.32, samples=2
iops : min= 0, max= 496, avg=348.50, stdev=208.60, samples=2
lat (nsec) : 750=35.07%, 1000=28.45%
lat (usec) : 2=12.71%, 4=0.05%, 10=0.53%, 20=0.29%, 50=0.14%
lat (usec) : 100=0.06%, 250=0.07%, 500=0.41%, 750=1.13%, 1000=2.65%
lat (msec) : 2=3.94%, 4=2.52%, 10=1.49%, 20=1.02%, 50=3.88%
lat (msec) : 100=3.09%, 250=1.23%, 500=0.96%, 750=0.25%, 1000=0.05%
cpu : usr=0.12%, sys=0.42%, ctx=18819, majf=0, minf=27
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=41856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=34, BW=2226KiB/s (2280kB/s)(261MiB/120024msec)
slat (nsec): min=0, max=1469.5M, avg=485436.54, stdev=23708009.49
clat (nsec): min=0, max=1068.9M, avg=10575940.19, stdev=49916812.10
lat (nsec): min=0, max=1470.0M, avg=11062295.90, stdev=55220809.67
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 1012],
| 50.00th=[ 1160], 60.00th=[ 1272], 70.00th=[ 1480],
| 80.00th=[ 193536], 90.00th=[ 21102592], 95.00th=[ 57933824],
| 99.00th=[ 235929600], 99.50th=[ 362807296], 99.90th=[ 616562688],
| 99.95th=[ 734003200], 99.99th=[1069547520]
bw ( KiB/s): min= 0, max= 2374, per=9.07%, avg=2226.00, stdev=209.30, samples=2
iops : min= 0, max= 37, avg=34.50, stdev= 3.54, samples=2
write: IOPS=36, BW=2331KiB/s (2387kB/s)(273MiB/120024msec)
slat (nsec): min=0, max=6481.8k, avg=159585.38, stdev=274880.69
clat (nsec): min=0, max=1784.7M, avg=99395294.56, stdev=159205507.24
lat (nsec): min=0, max=1784.7M, avg=99556016.47, stdev=159164612.17
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1549],
| 20.00th=[ 2343], 30.00th=[ 4228], 40.00th=[ 10683],
| 50.00th=[ 54264], 60.00th=[ 81265], 70.00th=[ 106431],
| 80.00th=[ 131597], 90.00th=[ 270533], 95.00th=[ 442500],
| 99.00th=[ 784335], 99.50th=[ 918553], 99.90th=[1384121],
| 99.95th=[1468007], 99.99th=[1786774]
bw ( KiB/s): min= 0, max= 2476, per=45.53%, avg=2328.50, stdev=208.60, samples=2
iops : min= 0, max= 38, avg=36.00, stdev= 2.83, samples=2
lat (nsec) : 750=7.96%, 1000=11.17%
lat (usec) : 2=16.96%, 4=0.06%, 10=0.27%, 20=0.83%, 50=0.71%
lat (usec) : 100=0.70%, 250=0.57%, 500=0.54%, 750=0.55%, 1000=1.05%
lat (msec) : 2=9.34%, 4=7.48%, 10=5.65%, 20=1.94%, 50=5.16%
lat (msec) : 100=11.00%, 250=12.30%, 500=3.83%, 750=1.32%, 1000=0.49%
cpu : usr=0.04%, sys=0.22%, ctx=10113, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4175,4368,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.0MiB/s (25.1MB/s), 2226KiB/s-21.8MiB/s (2280kB/s-22.9MB/s), io=2876MiB (3016MB), run=120001-120024msec
WRITE: bw=5114KiB/s (5237kB/s), 2331KiB/s-2784KiB/s (2387kB/s-2851kB/s), io=600MiB (629MB), run=120024-120043msec
Disk stats (read/write):
rbd2: ios=11019/10797, merge=0/1, ticks=541896/812590, in_queue=1343285, util=15.83%
<example-benchmark-fio-server-752-benchmark-kfgvn>
randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
write: IOPS=38, BW=2456KiB/s (2515kB/s)(288MiB/120075msec)
slat (nsec): min=0, max=486355k, avg=347151.01, stdev=8370452.54
clat (nsec): min=0, max=1379.9M, avg=104506667.37, stdev=171039013.89
lat (nsec): min=0, max=1379.0M, avg=104854912.10, stdev=171182329.65
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1385],
| 20.00th=[ 1860], 30.00th=[ 2835], 40.00th=[ 5735],
| 50.00th=[ 19006], 60.00th=[ 63701], 70.00th=[ 99091],
| 80.00th=[ 154141], 90.00th=[ 346031], 95.00th=[ 501220],
| 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[1132463],
| 99.95th=[1300235], 99.99th=[1384121]
bw ( KiB/s): min= 0, max= 3208, per=55.27%, avg=2455.00, stdev=1064.90, samples=2
iops : min= 0, max= 50, avg=38.00, stdev=16.97, samples=2
lat (usec) : 500=0.07%, 750=0.02%, 1000=1.13%
lat (msec) : 2=21.35%, 4=13.07%, 10=9.29%, 20=5.52%, 50=5.91%
lat (msec) : 100=13.77%, 250=15.44%, 500=9.38%, 750=4.13%, 1000=0.83%
cpu : usr=0.05%, sys=0.16%, ctx=8673, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4605,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2521MiB/120024msec)
slat (nsec): min=0, max=476699k, avg=39713.39, stdev=2703973.90
clat (nsec): min=0, max=1301.7M, avg=11885616.09, stdev=61578485.33
lat (nsec): min=0, max=1301.7M, avg=11926172.49, stdev=61648490.33
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
| 50.00th=[ 828], 60.00th=[ 876], 70.00th=[ 996],
| 80.00th=[ 1432], 90.00th=[ 2670592], 95.00th=[ 53215232],
| 99.00th=[ 371195904], 99.50th=[ 476053504], 99.90th=[ 675282944],
| 99.95th=[ 742391808], 99.99th=[1019215872]
bw ( KiB/s): min= 0, max=25368, per=91.91%, avg=21512.50, stdev=5452.50, samples=2
iops : min= 0, max= 396, avg=335.50, stdev=85.56, samples=2
lat (nsec) : 750=36.50%, 1000=33.72%
lat (usec) : 2=11.17%, 4=0.10%, 10=0.82%, 20=0.63%, 50=0.14%
lat (usec) : 100=0.07%, 250=0.05%, 500=0.21%, 750=0.93%, 1000=1.97%
lat (msec) : 2=2.80%, 4=1.64%, 10=0.82%, 20=0.49%, 50=2.68%
lat (msec) : 100=2.35%, 250=1.38%, 500=1.12%, 750=0.36%, 1000=0.04%
cpu : usr=0.11%, sys=0.37%, ctx=13493, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40363,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:55 2019
read: IOPS=29, BW=1897KiB/s (1943kB/s)(222MiB/120034msec)
slat (nsec): min=0, max=394240k, avg=221723.67, stdev=8238630.85
clat (nsec): min=0, max=533183k, avg=8440051.27, stdev=43256219.94
lat (nsec): min=0, max=533276k, avg=8662693.17, stdev=44003117.30
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 772], 30.00th=[ 860], 40.00th=[ 956],
| 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1432],
| 80.00th=[ 53504], 90.00th=[ 1630208], 95.00th=[ 42205184],
| 99.00th=[250609664], 99.50th=[346030080], 99.90th=[488636416],
| 99.95th=[530579456], 99.99th=[534773760]
bw ( KiB/s): min= 0, max= 2520, per=8.10%, avg=1896.50, stdev=881.76, samples=2
iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
write: IOPS=31, BW=1987KiB/s (2035kB/s)(233MiB/120034msec)
slat (nsec): min=0, max=4499.2k, avg=170586.55, stdev=218293.04
clat (nsec): min=0, max=1384.8M, avg=121032677.53, stdev=182906307.19
lat (nsec): min=0, max=1384.9M, avg=121204248.52, stdev=182846462.11
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
| 20.00th=[ 2089], 30.00th=[ 3621], 40.00th=[ 9765],
| 50.00th=[ 45351], 60.00th=[ 81265], 70.00th=[ 112722],
| 80.00th=[ 206570], 90.00th=[ 392168], 95.00th=[ 541066],
| 99.00th=[ 767558], 99.50th=[ 901776], 99.90th=[1052771],
| 99.95th=[1300235], 99.99th=[1384121]
bw ( KiB/s): min= 0, max= 2640, per=44.71%, avg=1986.00, stdev=924.90, samples=2
iops : min= 0, max= 41, avg=30.50, stdev=14.85, samples=2
lat (nsec) : 750=8.23%, 1000=12.94%
lat (usec) : 2=15.82%, 4=0.07%, 10=0.12%, 20=0.73%, 50=1.07%
lat (usec) : 100=1.07%, 250=0.65%, 500=0.91%, 750=0.58%, 1000=1.33%
lat (msec) : 2=10.42%, 4=6.85%, 10=4.79%, 20=2.93%, 50=4.44%
lat (msec) : 100=8.27%, 250=9.74%, 500=5.82%, 750=2.66%, 1000=0.52%
cpu : usr=0.06%, sys=0.16%, ctx=8298, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3559,3723,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=22.9MiB/s (23.0MB/s), 1897KiB/s-21.0MiB/s (1943kB/s-22.0MB/s), io=2744MiB (2877MB), run=120024-120034msec
WRITE: bw=4443KiB/s (4549kB/s), 1987KiB/s-2456KiB/s (2035kB/s-2515kB/s), io=521MiB (546MB), run=120034-120075msec
Disk stats (read/write):
rbd272: ios=7654/8582, merge=0/1, ticks=537074/870601, in_queue=1399336, util=11.51%
<example-benchmark-fio-server-427-benchmark-hzqrn>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=55, BW=3533KiB/s (3617kB/s)(414MiB/120097msec)
slat (nsec): min=0, max=456194k, avg=355259.52, stdev=8398375.70
clat (nsec): min=0, max=991198k, avg=72238614.86, stdev=141200455.04
lat (nsec): min=0, max=991288k, avg=72595361.15, stdev=141328353.19
clat percentiles (usec):
| 1.00th=[ 1004], 5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1795],
| 30.00th=[ 2409], 40.00th=[ 3654], 50.00th=[ 6521], 60.00th=[ 16450],
| 70.00th=[ 69731], 80.00th=[104334], 90.00th=[204473], 95.00th=[392168],
| 99.00th=[708838], 99.50th=[792724], 99.90th=[926942], 99.95th=[952108],
| 99.99th=[994051]
bw ( KiB/s): min= 0, max= 4756, per=56.38%, avg=3533.00, stdev=1729.58, samples=2
iops : min= 0, max= 74, avg=55.00, stdev=26.87, samples=2
lat (usec) : 500=0.02%, 750=0.03%, 1000=0.92%
lat (msec) : 2=23.29%, 4=17.60%, 10=13.67%, 20=5.77%, 50=4.87%
lat (msec) : 100=12.65%, 250=12.93%, 500=4.95%, 750=2.69%, 1000=0.66%
cpu : usr=0.08%, sys=0.23%, ctx=12600, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6626,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=336, BW=21.0MiB/s (22.1MB/s)(2526MiB/120015msec)
slat (nsec): min=0, max=362838k, avg=45577.70, stdev=2841029.73
clat (nsec): min=0, max=1029.8M, avg=11837195.23, stdev=59499909.85
lat (nsec): min=0, max=1029.9M, avg=11883621.69, stdev=59574212.09
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
| 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1080],
| 80.00th=[ 60672], 90.00th=[ 9240576], 95.00th=[ 54788096],
| 99.00th=[346030080], 99.50th=[476053504], 99.90th=[692060160],
| 99.95th=[750780416], 99.99th=[943718400]
bw ( KiB/s): min= 0, max=28561, per=89.07%, avg=21554.00, stdev=9909.39, samples=2
iops : min= 0, max= 446, avg=336.50, stdev=154.86, samples=2
lat (nsec) : 750=39.64%, 1000=28.22%
lat (usec) : 2=10.44%, 4=0.04%, 10=0.88%, 20=0.50%, 50=0.24%
lat (usec) : 100=0.10%, 250=0.06%, 500=0.15%, 750=0.98%, 1000=2.14%
lat (msec) : 2=3.42%, 4=2.12%, 10=1.14%, 20=0.74%, 50=3.59%
lat (msec) : 100=2.99%, 250=1.23%, 500=0.95%, 750=0.37%, 1000=0.05%
cpu : usr=0.12%, sys=0.38%, ctx=16094, majf=0, minf=102
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=41, BW=2654KiB/s (2718kB/s)(311MiB/120055msec)
slat (nsec): min=0, max=238280k, avg=92609.11, stdev=3549619.81
clat (nsec): min=0, max=774600k, avg=8859266.13, stdev=41684463.38
lat (nsec): min=0, max=774650k, avg=8952847.07, stdev=41829723.60
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 764],
| 20.00th=[ 844], 30.00th=[ 924], 40.00th=[ 1048],
| 50.00th=[ 1192], 60.00th=[ 1320], 70.00th=[ 1608],
| 80.00th=[ 370688], 90.00th=[ 9895936], 95.00th=[ 52690944],
| 99.00th=[191889408], 99.50th=[312475648], 99.90th=[476053504],
| 99.95th=[658505728], 99.99th=[775946240]
bw ( KiB/s): min= 0, max= 3376, per=10.97%, avg=2655.00, stdev=1019.65, samples=2
iops : min= 0, max= 52, avg=41.00, stdev=15.56, samples=2
write: IOPS=42, BW=2735KiB/s (2801kB/s)(321MiB/120055msec)
slat (nsec): min=0, max=634946k, avg=345574.69, stdev=9626503.84
clat (nsec): min=0, max=1219.3M, avg=84920091.81, stdev=152543276.10
lat (nsec): min=0, max=1219.3M, avg=85266747.75, stdev=152708991.55
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
| 20.00th=[ 1942], 30.00th=[ 2933], 40.00th=[ 4817],
| 50.00th=[ 10290], 60.00th=[ 45876], 70.00th=[ 84411],
| 80.00th=[ 115868], 90.00th=[ 254804], 95.00th=[ 467665],
| 99.00th=[ 725615], 99.50th=[ 801113], 99.90th=[ 952108],
| 99.95th=[ 985662], 99.99th=[1216349]
bw ( KiB/s): min= 0, max= 3446, per=43.63%, avg=2734.00, stdev=1006.92, samples=2
iops : min= 0, max= 53, avg=42.00, stdev=15.56, samples=2
lat (nsec) : 750=4.32%, 1000=13.77%
lat (usec) : 2=17.31%, 4=0.07%, 10=0.30%, 20=0.92%, 50=0.85%
lat (usec) : 100=0.87%, 250=0.72%, 500=0.62%, 750=0.53%, 1000=1.40%
lat (msec) : 2=11.97%, 4=8.73%, 10=7.17%, 20=3.20%, 50=4.70%
lat (msec) : 100=9.37%, 250=7.66%, 500=3.41%, 750=1.75%, 1000=0.36%
cpu : usr=0.07%, sys=0.25%, ctx=11962, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4980,5128,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.6MiB/s (24.8MB/s), 2654KiB/s-21.0MiB/s (2718kB/s-22.1MB/s), io=2837MiB (2975MB), run=120015-120055msec
WRITE: bw=6267KiB/s (6417kB/s), 2735KiB/s-3533KiB/s (2801kB/s-3617kB/s), io=735MiB (771MB), run=120055-120097msec
Disk stats (read/write):
rbd131: ios=9638/11955, merge=0/1, ticks=544140/818314, in_queue=1351514, util=15.67%
<example-benchmark-fio-server-731-benchmark-2p24w>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
write: IOPS=46, BW=2954KiB/s (3025kB/s)(346MiB/120061msec)
slat (nsec): min=0, max=750892k, avg=445086.93, stdev=11721042.12
clat (nsec): min=0, max=1628.2M, avg=86222905.58, stdev=150130186.58
lat (nsec): min=0, max=1628.3M, avg=86669252.57, stdev=150470281.13
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1500],
| 20.00th=[ 2147], 30.00th=[ 3228], 40.00th=[ 5604],
| 50.00th=[ 13173], 60.00th=[ 55837], 70.00th=[ 93848],
| 80.00th=[ 123208], 90.00th=[ 240124], 95.00th=[ 421528],
| 99.00th=[ 708838], 99.50th=[ 801113], 99.90th=[1098908],
| 99.95th=[1115685], 99.99th=[1635779]
bw ( KiB/s): min= 0, max= 4555, per=52.95%, avg=2953.00, stdev=2265.57, samples=2
iops : min= 0, max= 71, avg=46.00, stdev=35.36, samples=2
lat (usec) : 100=0.02%, 250=0.04%, 500=0.04%, 750=0.07%, 1000=1.39%
lat (msec) : 2=16.67%, 4=16.38%, 10=13.04%, 20=4.75%, 50=6.41%
lat (msec) : 100=13.20%, 250=18.62%, 500=5.99%, 750=2.67%, 1000=0.58%
cpu : usr=0.05%, sys=0.20%, ctx=10208, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5538,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
read: IOPS=529, BW=33.0MiB/s (34.7MB/s)(3966MiB/120019msec)
slat (nsec): min=0, max=482645k, avg=34306.21, stdev=2614937.41
clat (nsec): min=0, max=1336.0M, avg=7539479.66, stdev=48759461.40
lat (nsec): min=0, max=1336.0M, avg=7574591.66, stdev=48839095.66
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 804], 60.00th=[ 868], 70.00th=[ 964],
| 80.00th=[ 8640], 90.00th=[ 1089536], 95.00th=[ 32636928],
| 99.00th=[ 217055232], 99.50th=[ 404750336], 99.90th=[ 633339904],
| 99.95th=[ 742391808], 99.99th=[1149239296]
bw ( KiB/s): min= 0, max=46913, per=93.04%, avg=33844.00, stdev=18482.36, samples=2
iops : min= 0, max= 735, avg=529.50, stdev=290.62, samples=2
lat (nsec) : 750=44.39%, 1000=26.82%
lat (usec) : 2=7.58%, 4=0.06%, 10=4.95%, 20=2.19%, 50=0.69%
lat (usec) : 100=0.15%, 250=0.06%, 500=0.17%, 750=0.85%, 1000=1.70%
lat (msec) : 2=2.34%, 4=1.32%, 10=0.59%, 20=0.39%, 50=2.28%
lat (msec) : 100=1.84%, 250=0.73%, 500=0.67%, 750=0.20%, 1000=0.03%
cpu : usr=0.16%, sys=0.53%, ctx=16677, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=63607,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
read: IOPS=39, BW=2534KiB/s (2595kB/s)(297MiB/120003msec)
slat (nsec): min=0, max=2602.2k, avg=26859.27, stdev=68750.48
clat (nsec): min=0, max=750910k, avg=6636280.46, stdev=36858791.12
lat (nsec): min=0, max=750930k, avg=6664125.19, stdev=36867160.13
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
| 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 980],
| 50.00th=[ 1128], 60.00th=[ 1272], 70.00th=[ 1592],
| 80.00th=[ 84480], 90.00th=[ 1613824], 95.00th=[ 38535168],
| 99.00th=[160432128], 99.50th=[295698432], 99.90th=[480247808],
| 99.95th=[488636416], 99.99th=[750780416]
bw ( KiB/s): min= 0, max= 3603, per=6.97%, avg=2534.00, stdev=1511.79, samples=2
iops : min= 0, max= 56, avg=39.00, stdev=24.04, samples=2
write: IOPS=40, BW=2625KiB/s (2688kB/s)(308MiB/120003msec)
slat (nsec): min=0, max=750900k, avg=473560.27, stdev=12692496.43
clat (nsec): min=0, max=1632.7M, avg=90831011.95, stdev=157168249.28
lat (nsec): min=0, max=1632.8M, avg=91305477.41, stdev=157556825.13
clat percentiles (usec):
| 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
| 20.00th=[ 2147], 30.00th=[ 3326], 40.00th=[ 5866],
| 50.00th=[ 15795], 60.00th=[ 62129], 70.00th=[ 96994],
| 80.00th=[ 127402], 90.00th=[ 256902], 95.00th=[ 442500],
| 99.00th=[ 742392], 99.50th=[ 868221], 99.90th=[1233126],
| 99.95th=[1283458], 99.99th=[1635779]
bw ( KiB/s): min= 0, max= 3730, per=47.02%, avg=2622.50, stdev=1566.24, samples=2
iops : min= 0, max= 58, avg=40.50, stdev=24.75, samples=2
lat (nsec) : 750=5.76%, 1000=14.44%
lat (usec) : 2=15.05%, 4=0.09%, 10=0.52%, 20=1.30%, 50=1.23%
lat (usec) : 100=1.25%, 250=1.03%, 500=1.04%, 750=0.69%, 1000=1.35%
lat (msec) : 2=10.10%, 4=8.67%, 10=6.74%, 20=2.49%, 50=4.65%
lat (msec) : 100=8.20%, 250=9.94%, 500=3.54%, 750=1.44%, 1000=0.35%
cpu : usr=0.06%, sys=0.23%, ctx=11015, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4755,4919,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=35.5MiB/s (37.2MB/s), 2534KiB/s-33.0MiB/s (2595kB/s-34.7MB/s), io=4263MiB (4470MB), run=120003-120019msec
WRITE: bw=5577KiB/s (5711kB/s), 2625KiB/s-2954KiB/s (2688kB/s-3025kB/s), io=654MiB (686MB), run=120003-120061msec
Disk stats (read/write):
rbd178: ios=9145/10566, merge=0/3, ticks=516068/791944, in_queue=1297922, util=13.11%
Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1375M,w=1387M][r=21.5k,w=21.7k IOPS][eta 00m:54s] Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1410M,w=1427M][r=22.0k,w=22.3k IOPS][eta 00m:53s] <example-benchmark-fio-server-794-benchmark-xkzrq>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:57 2019
write: IOPS=39, BW=2511KiB/s (2572kB/s)(294MiB/120011msec)
slat (nsec): min=0, max=830857k, avg=575567.16, stdev=16158635.08
clat (nsec): min=0, max=1469.7M, avg=101665439.36, stdev=163496335.02
lat (nsec): min=0, max=1469.8M, avg=102242584.08, stdev=164095874.59
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1270], 10.00th=[ 1598],
| 20.00th=[ 2245], 30.00th=[ 3720], 40.00th=[ 9503],
| 50.00th=[ 42730], 60.00th=[ 72877], 70.00th=[ 102237],
| 80.00th=[ 143655], 90.00th=[ 291505], 95.00th=[ 438305],
| 99.00th=[ 801113], 99.50th=[ 918553], 99.90th=[1434452],
| 99.95th=[1468007], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 3129, per=48.79%, avg=2509.00, stdev=876.81, samples=2
iops : min= 0, max= 48, avg=38.50, stdev=13.44, samples=2
lat (usec) : 2=0.02%, 4=0.02%, 500=0.02%, 750=0.04%, 1000=0.70%
lat (msec) : 2=15.68%, 4=14.53%, 10=9.31%, 20=4.12%, 50=7.39%
lat (msec) : 100=17.68%, 250=18.25%, 500=8.35%, 750=2.57%, 1000=1.13%
cpu : usr=0.06%, sys=0.17%, ctx=8853, majf=0, minf=26
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4706,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
read: IOPS=321, BW=20.1MiB/s (21.1MB/s)(2412MiB/120011msec)
slat (nsec): min=0, max=593474k, avg=62460.39, stdev=4735684.38
clat (nsec): min=0, max=1023.2M, avg=12403573.49, stdev=60386500.08
lat (nsec): min=0, max=1023.3M, avg=12466969.14, stdev=60576303.42
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
| 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 996],
| 80.00th=[ 1528], 90.00th=[ 11075584], 95.00th=[ 57409536],
| 99.00th=[346030080], 99.50th=[467664896], 99.90th=[700448768],
| 99.95th=[775946240], 99.99th=[952107008]
bw ( KiB/s): min= 0, max=27268, per=89.00%, avg=20583.50, stdev=9453.31, samples=2
iops : min= 0, max= 426, avg=321.50, stdev=147.79, samples=2
lat (nsec) : 750=42.48%, 1000=27.64%
lat (usec) : 2=10.36%, 4=0.04%, 10=0.65%, 20=0.38%, 50=0.32%
lat (usec) : 100=0.13%, 250=0.04%, 500=0.20%, 750=0.85%, 1000=1.83%
lat (msec) : 2=2.50%, 4=1.57%, 10=0.91%, 20=0.91%, 50=3.52%
lat (msec) : 100=2.58%, 250=1.53%, 500=1.16%, 750=0.33%, 1000=0.06%
cpu : usr=0.11%, sys=0.36%, ctx=13842, majf=0, minf=109
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=38631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
read: IOPS=39, BW=2544KiB/s (2605kB/s)(298MiB/120003msec)
slat (nsec): min=0, max=578056k, avg=151252.96, stdev=8368754.74
clat (nsec): min=0, max=1230.4M, avg=8238059.24, stdev=45356635.31
lat (nsec): min=0, max=1230.4M, avg=8390234.99, stdev=46106318.45
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
| 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 988],
| 50.00th=[ 1144], 60.00th=[ 1272], 70.00th=[ 1560],
| 80.00th=[ 232448], 90.00th=[ 3686400], 95.00th=[ 41156608],
| 99.00th=[ 221249536], 99.50th=[ 354418688], 99.90th=[ 480247808],
| 99.95th=[ 801112064], 99.99th=[1233125376]
bw ( KiB/s): min= 0, max= 3363, per=11.00%, avg=2543.00, stdev=1159.66, samples=2
iops : min= 0, max= 52, avg=39.00, stdev=18.38, samples=2
write: IOPS=41, BW=2632KiB/s (2695kB/s)(308MiB/120003msec)
slat (nsec): min=0, max=451778k, avg=272568.46, stdev=6436634.86
clat (nsec): min=0, max=1469.7M, avg=88949844.53, stdev=152195506.47
lat (nsec): min=0, max=1469.8M, avg=89223694.54, stdev=152274142.94
clat percentiles (usec):
| 1.00th=[ 1045], 5.00th=[ 1401], 10.00th=[ 1729],
| 20.00th=[ 2540], 30.00th=[ 4015], 40.00th=[ 7308],
| 50.00th=[ 19792], 60.00th=[ 54789], 70.00th=[ 87557],
| 80.00th=[ 124257], 90.00th=[ 265290], 95.00th=[ 421528],
| 99.00th=[ 742392], 99.50th=[ 876610], 99.90th=[ 952108],
| 99.95th=[1434452], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 3460, per=51.16%, avg=2630.50, stdev=1173.09, samples=2
iops : min= 0, max= 54, avg=41.00, stdev=18.38, samples=2
lat (nsec) : 750=6.93%, 1000=13.11%
lat (usec) : 2=15.45%, 4=0.07%, 10=0.41%, 20=0.79%, 50=0.92%
lat (usec) : 100=0.97%, 250=0.75%, 500=0.82%, 750=0.54%, 1000=1.22%
lat (msec) : 2=8.42%, 4=9.14%, 10=7.65%, 20=3.96%, 50=5.69%
lat (msec) : 100=9.08%, 250=8.22%, 500=4.08%, 750=1.28%, 1000=0.48%
cpu : usr=0.07%, sys=0.23%, ctx=11104, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4771,4932,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=22.6MiB/s (23.7MB/s), 2544KiB/s-20.1MiB/s (2605kB/s-21.1MB/s), io=2711MiB (2842MB), run=120003-120011msec
WRITE: bw=5143KiB/s (5266kB/s), 2511KiB/s-2632KiB/s (2572kB/s-2695kB/s), io=603MiB (632MB), run=120003-120011msec
Disk stats (read/write):
rbd206: ios=9196/10145, merge=0/2, ticks=549346/854017, in_queue=1393497, util=13.88%
<example-benchmark-fio-server-694-benchmark-wvfzs>
randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
write: IOPS=40, BW=2571KiB/s (2633kB/s)(307MiB/122324msec)
slat (nsec): min=0, max=25427k, avg=195885.07, stdev=562273.47
clat (nsec): min=0, max=8959.5M, avg=99109794.52, stdev=322776851.58
lat (nsec): min=0, max=8959.6M, avg=99307113.25, stdev=322783856.59
clat percentiles (usec):
| 1.00th=[ 848], 5.00th=[ 1827], 10.00th=[ 2802],
| 20.00th=[ 4293], 30.00th=[ 6718], 40.00th=[ 10421],
| 50.00th=[ 15533], 60.00th=[ 27395], 70.00th=[ 54264],
| 80.00th=[ 130548], 90.00th=[ 287310], 95.00th=[ 429917],
| 99.00th=[ 717226], 99.50th=[1317012], 99.90th=[6878659],
| 99.95th=[8556381], 99.99th=[8925479]
bw ( KiB/s): min= 0, max= 3233, per=47.42%, avg=2618.50, stdev=869.03, samples=2
iops : min= 0, max= 50, avg=40.50, stdev=13.44, samples=2
lat (usec) : 2=0.02%, 4=0.02%, 100=0.02%, 250=0.02%, 500=0.06%
lat (usec) : 750=0.37%, 1000=1.06%
lat (msec) : 2=4.28%, 4=11.89%, 10=21.32%, 20=15.64%, 50=14.05%
lat (msec) : 100=8.90%, 250=10.59%, 500=8.33%, 750=2.61%, 1000=0.20%
cpu : usr=0.03%, sys=0.15%, ctx=7781, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4911,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
read: IOPS=68, BW=4371KiB/s (4476kB/s)(513MiB/120237msec)
slat (nsec): min=0, max=9176.8k, avg=52739.78, stdev=188486.51
clat (nsec): min=0, max=1825.5M, avg=58615474.01, stdev=197991591.34
lat (nsec): min=0, max=1825.5M, avg=58669431.87, stdev=198008807.92
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 772],
| 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 1032],
| 50.00th=[ 1256], 60.00th=[ 1640], 70.00th=[ 1417216],
| 80.00th=[ 3424256], 90.00th=[ 49020928], 95.00th=[ 599785472],
| 99.00th=[ 952107008], 99.50th=[1069547520], 99.90th=[1350565888],
| 99.95th=[1518338048], 99.99th=[1820327936]
bw ( KiB/s): min= 0, max= 4448, per=60.15%, avg=4376.50, stdev=101.12, samples=2
iops : min= 0, max= 69, avg=68.00, stdev= 1.41, samples=2
lat (nsec) : 750=6.54%, 1000=32.33%
lat (usec) : 2=21.82%, 4=0.22%, 10=0.11%, 20=0.52%, 50=0.17%
lat (usec) : 100=0.19%, 250=0.10%, 500=0.06%, 750=0.82%, 1000=3.16%
lat (msec) : 2=7.88%, 4=7.27%, 10=4.92%, 20=2.01%, 50=1.99%
lat (msec) : 100=0.97%, 250=0.83%, 500=2.22%, 750=3.12%, 1000=2.05%
cpu : usr=0.04%, sys=0.11%, ctx=6246, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=8209,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:57 2019
read: IOPS=45, BW=2911KiB/s (2981kB/s)(341MiB/120007msec)
slat (nsec): min=0, max=3786.0k, avg=42845.43, stdev=115101.89
clat (nsec): min=0, max=809090k, avg=19029910.56, stdev=72180861.64
lat (nsec): min=0, max=809121k, avg=19073885.97, stdev=72190890.86
clat percentiles (nsec):
| 1.00th=[ 724], 5.00th=[ 740], 10.00th=[ 748],
| 20.00th=[ 868], 30.00th=[ 1004], 40.00th=[ 1144],
| 50.00th=[ 1304], 60.00th=[ 52480], 70.00th=[ 1105920],
| 80.00th=[ 3227648], 90.00th=[ 43778048], 95.00th=[ 87556096],
| 99.00th=[446693376], 99.50th=[509607936], 99.90th=[734003200],
| 99.95th=[784334848], 99.99th=[809500672]
bw ( KiB/s): min= 0, max= 4526, per=39.99%, avg=2909.50, stdev=2286.08, samples=2
iops : min= 0, max= 70, avg=45.00, stdev=35.36, samples=2
write: IOPS=46, BW=3008KiB/s (3081kB/s)(353MiB/120007msec)
slat (nsec): min=0, max=5596.6k, avg=184642.48, stdev=236894.22
clat (nsec): min=0, max=852576k, avg=66695651.86, stdev=130745032.75
lat (nsec): min=0, max=852605k, avg=66881887.03, stdev=130698526.14
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1369], 20.00th=[ 1696],
| 30.00th=[ 2212], 40.00th=[ 3163], 50.00th=[ 5145], 60.00th=[ 10945],
| 70.00th=[ 49546], 80.00th=[102237], 90.00th=[200279], 95.00th=[383779],
| 99.00th=[616563], 99.50th=[683672], 99.90th=[750781], 99.95th=[826278],
| 99.99th=[851444]
bw ( KiB/s): min= 0, max= 4664, per=54.45%, avg=3007.00, stdev=2343.35, samples=2
iops : min= 0, max= 72, avg=46.50, stdev=36.06, samples=2
lat (nsec) : 750=4.73%, 1000=9.89%
lat (usec) : 2=14.22%, 4=0.05%, 10=0.07%, 20=0.32%, 50=0.23%
lat (usec) : 100=0.32%, 250=0.35%, 500=0.22%, 750=1.08%, 1000=2.54%
lat (msec) : 2=17.03%, 4=12.00%, 10=8.42%, 20=3.30%, 50=5.66%
lat (msec) : 100=6.99%, 250=7.36%, 500=3.60%, 750=1.53%, 1000=0.12%
cpu : usr=0.09%, sys=0.25%, ctx=14803, majf=0, minf=37
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5457,5640,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=7277KiB/s (7451kB/s), 2911KiB/s-4371KiB/s (2981kB/s-4476kB/s), io=854MiB (896MB), run=120007-120237msec
WRITE: bw=5522KiB/s (5655kB/s), 2571KiB/s-3008KiB/s (2633kB/s-3081kB/s), io=660MiB (692MB), run=120007-122324msec
Disk stats (read/write):
rbd204: ios=5634/12337, merge=0/3, ticks=642225/878157, in_queue=1511367, util=4.27%
<example-benchmark-fio-server-612-benchmark-2dg6w>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=27, BW=1758KiB/s (1800kB/s)(207MiB/120534msec)
slat (nsec): min=0, max=4257.1M, avg=2010805.23, stdev=76515967.14
clat (nsec): min=0, max=6824.8M, avg=143809839.17, stdev=306619330.65
lat (nsec): min=0, max=6833.9M, avg=145820705.46, stdev=319679458.15
clat percentiles (usec):
| 1.00th=[ 1221], 5.00th=[ 1663], 10.00th=[ 2114],
| 20.00th=[ 3032], 30.00th=[ 4178], 40.00th=[ 6063],
| 50.00th=[ 9634], 60.00th=[ 16909], 70.00th=[ 88605],
| 80.00th=[ 333448], 90.00th=[ 492831], 95.00th=[ 616563],
| 99.00th=[ 809501], 99.50th=[1035994], 99.90th=[3506439],
| 99.95th=[6811550], 99.99th=[6811550]
bw ( KiB/s): min= 0, max= 2243, per=33.59%, avg=1763.00, stdev=678.82, samples=2
iops : min= 0, max= 35, avg=27.50, stdev=10.61, samples=2
lat (usec) : 500=0.03%, 750=0.03%, 1000=0.18%
lat (msec) : 2=8.41%, 4=19.78%, 10=22.32%, 20=11.82%, 50=6.23%
lat (msec) : 100=1.45%, 250=5.47%, 500=14.45%, 750=8.50%, 1000=0.88%
cpu : usr=0.02%, sys=0.12%, ctx=5571, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3307,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=136, BW=8764KiB/s (8975kB/s)(1032MiB/120562msec)
slat (nsec): min=0, max=12461k, avg=36945.42, stdev=180100.44
clat (nsec): min=0, max=1972.8M, avg=29204481.49, stdev=143415318.41
lat (nsec): min=0, max=1972.8M, avg=29242530.82, stdev=143430361.26
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 756],
| 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
| 50.00th=[ 900], 60.00th=[ 1032], 70.00th=[ 1320],
| 80.00th=[ 995328], 90.00th=[ 4685824], 95.00th=[ 51642368],
| 99.00th=[ 826277888], 99.50th=[ 952107008], 99.90th=[1333788672],
| 99.95th=[1535115264], 99.99th=[1837105152]
bw ( KiB/s): min= 0, max= 9481, per=72.35%, avg=8802.50, stdev=959.54, samples=2
iops : min= 0, max= 148, avg=137.00, stdev=15.56, samples=2
lat (nsec) : 750=9.74%, 1000=49.14%
lat (usec) : 2=17.42%, 4=0.25%, 10=0.19%, 20=0.59%, 50=0.16%
lat (usec) : 100=0.12%, 250=0.12%, 500=0.13%, 750=0.43%, 1000=1.79%
lat (msec) : 2=4.74%, 4=4.43%, 10=3.01%, 20=1.31%, 50=1.40%
lat (msec) : 100=0.57%, 250=0.58%, 500=1.05%, 750=1.38%, 1000=1.13%
cpu : usr=0.06%, sys=0.18%, ctx=7446, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=16507,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=53, BW=3419KiB/s (3501kB/s)(401MiB/120002msec)
slat (nsec): min=0, max=1954.4k, avg=31149.84, stdev=77107.69
clat (nsec): min=0, max=734117k, avg=11037698.80, stdev=54963807.10
lat (nsec): min=0, max=734145k, avg=11069791.59, stdev=54971279.93
clat percentiles (nsec):
| 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
| 20.00th=[ 812], 30.00th=[ 908], 40.00th=[ 1032],
| 50.00th=[ 1176], 60.00th=[ 1304], 70.00th=[ 1816],
| 80.00th=[ 856064], 90.00th=[ 5275648], 95.00th=[ 51118080],
| 99.00th=[320864256], 99.50th=[455081984], 99.90th=[616562688],
| 99.95th=[650117120], 99.99th=[734003200]
bw ( KiB/s): min= 0, max= 4692, per=28.10%, avg=3418.50, stdev=1801.00, samples=2
iops : min= 0, max= 73, avg=53.00, stdev=28.28, samples=2
write: IOPS=54, BW=3508KiB/s (3592kB/s)(411MiB/120002msec)
slat (nsec): min=0, max=10682k, avg=192837.44, stdev=258303.43
clat (nsec): min=0, max=1267.3M, avg=62261381.68, stdev=127104471.59
lat (nsec): min=0, max=1267.3M, avg=62455492.49, stdev=127055529.35
clat percentiles (usec):
| 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1401],
| 20.00th=[ 1795], 30.00th=[ 2376], 40.00th=[ 3458],
| 50.00th=[ 5997], 60.00th=[ 14484], 70.00th=[ 32900],
| 80.00th=[ 99091], 90.00th=[ 187696], 95.00th=[ 325059],
| 99.00th=[ 675283], 99.50th=[ 750781], 99.90th=[ 901776],
| 99.95th=[ 977273], 99.99th=[1266680]
bw ( KiB/s): min= 0, max= 4778, per=66.77%, avg=3505.00, stdev=1800.29, samples=2
iops : min= 0, max= 74, avg=54.00, stdev=28.28, samples=2
lat (nsec) : 750=5.97%, 1000=12.84%
lat (usec) : 2=15.87%, 4=0.08%, 10=0.23%, 20=0.78%, 50=0.68%
lat (usec) : 100=0.62%, 250=0.69%, 500=0.44%, 750=0.65%, 1000=1.93%
lat (msec) : 2=14.01%, 4=11.19%, 10=7.22%, 20=4.84%, 50=5.47%
lat (msec) : 100=5.08%, 250=7.06%, 500=3.13%, 750=0.98%, 1000=0.25%
cpu : usr=0.09%, sys=0.31%, ctx=15021, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6410,6574,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=11.9MiB/s (12.5MB/s), 3419KiB/s-8764KiB/s (3501kB/s-8975kB/s), io=1433MiB (1502MB), run=120002-120562msec
WRITE: bw=5250KiB/s (5376kB/s), 1758KiB/s-3508KiB/s (1800kB/s-3592kB/s), io=618MiB (648MB), run=120002-120534msec
Disk stats (read/write):
rbd279: ios=5460/10356, merge=0/3, ticks=596117/878665, in_queue=1466829, util=4.57%
Jobs: 48 (f=45): [_(2),m(1)][99.8%][r=1436M,w=1434M][r=22.4k,w=22.4k IOPS][eta 00m:52s] <example-benchmark-fio-server-898-benchmark-2q9b7>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=13, BW=835KiB/s (855kB/s)(98.1MiB/120351msec)
slat (nsec): min=0, max=2022.3M, avg=2728134.72, stdev=58249237.85
clat (nsec): min=0, max=3275.6M, avg=304687648.20, stdev=456863340.63
lat (nsec): min=0, max=3275.7M, avg=307412545.76, stdev=460022219.28
clat percentiles (usec):
| 1.00th=[ 1303], 5.00th=[ 2024], 10.00th=[ 2704],
| 20.00th=[ 4490], 30.00th=[ 6915], 40.00th=[ 11338],
| 50.00th=[ 18744], 60.00th=[ 50070], 70.00th=[ 392168],
| 80.00th=[ 633340], 90.00th=[1010828], 95.00th=[1317012],
| 99.00th=[1635779], 99.50th=[1920992], 99.90th=[2231370],
| 99.95th=[3271558], 99.99th=[3271558]
bw ( KiB/s): min= 0, max= 1076, per=21.35%, avg=833.00, stdev=343.65, samples=2
iops : min= 0, max= 16, avg=12.50, stdev= 4.95, samples=2
lat (usec) : 1000=0.19%
lat (msec) : 2=4.53%, 4=12.95%, 10=19.85%, 20=13.85%, 50=8.74%
lat (msec) : 100=1.91%, 250=0.89%, 500=11.81%, 750=8.42%, 1000=6.76%
cpu : usr=0.02%, sys=0.07%, ctx=2848, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1567,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=307, BW=19.2MiB/s (20.1MB/s)(2315MiB/120566msec)
slat (nsec): min=0, max=1464.6M, avg=73420.58, stdev=8011473.67
clat (nsec): min=0, max=2370.1M, avg=12938782.83, stdev=99222958.48
lat (nsec): min=0, max=2370.1M, avg=13013350.53, stdev=99551960.23
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
| 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
| 80.00th=[ 1020], 90.00th=[ 11840], 95.00th=[ 2703360],
| 99.00th=[ 624951296], 99.50th=[ 792723456], 99.90th=[1199570944],
| 99.95th=[1384120320], 99.99th=[1937768448]
bw ( KiB/s): min= 0, max=26548, per=87.30%, avg=19751.50, stdev=9611.70, samples=2
iops : min= 0, max= 415, avg=308.50, stdev=150.61, samples=2
lat (nsec) : 750=12.45%, 1000=67.02%
lat (usec) : 2=9.06%, 4=0.23%, 10=0.46%, 20=2.23%, 50=0.58%
lat (usec) : 100=0.18%, 250=0.09%, 500=0.13%, 750=0.15%, 1000=0.44%
lat (msec) : 2=1.38%, 4=1.28%, 10=1.15%, 20=0.51%, 50=0.46%
lat (msec) : 100=0.24%, 250=0.24%, 500=0.46%, 750=0.65%, 1000=0.42%
cpu : usr=0.08%, sys=0.34%, ctx=5906, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=37091,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=46, BW=2977KiB/s (3048kB/s)(349MiB/120021msec)
slat (nsec): min=0, max=5457.3k, avg=23255.68, stdev=98205.93
clat (nsec): min=0, max=786199k, avg=7736512.34, stdev=47943367.55
lat (nsec): min=0, max=786220k, avg=7760701.96, stdev=47953450.62
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 724],
| 20.00th=[ 748], 30.00th=[ 828], 40.00th=[ 916],
| 50.00th=[ 1048], 60.00th=[ 1192], 70.00th=[ 1368],
| 80.00th=[ 19328], 90.00th=[ 978944], 95.00th=[ 24772608],
| 99.00th=[242221056], 99.50th=[404750336], 99.90th=[608174080],
| 99.95th=[700448768], 99.99th=[784334848]
bw ( KiB/s): min= 0, max= 4710, per=13.16%, avg=2976.50, stdev=2451.54, samples=2
iops : min= 0, max= 73, avg=46.00, stdev=38.18, samples=2
write: IOPS=48, BW=3076KiB/s (3150kB/s)(361MiB/120021msec)
slat (nsec): min=0, max=6098.2k, avg=209590.76, stdev=291288.10
clat (nsec): min=0, max=1088.8M, avg=75761080.52, stdev=143915707.64
lat (nsec): min=0, max=1088.8M, avg=75971772.86, stdev=143854208.57
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1287],
| 20.00th=[ 1663], 30.00th=[ 2278], 40.00th=[ 3556],
| 50.00th=[ 6980], 60.00th=[ 21890], 70.00th=[ 68682],
| 80.00th=[ 112722], 90.00th=[ 231736], 95.00th=[ 400557],
| 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 968885],
| 99.95th=[ 994051], 99.99th=[1082131]
bw ( KiB/s): min= 0, max= 4848, per=78.78%, avg=3074.00, stdev=2508.81, samples=2
iops : min= 0, max= 75, avg=47.50, stdev=38.89, samples=2
lat (nsec) : 750=10.49%, 1000=12.60%
lat (usec) : 2=14.41%, 4=0.04%, 10=0.41%, 20=1.55%, 50=1.02%
lat (usec) : 100=0.98%, 250=1.01%, 500=0.77%, 750=0.50%, 1000=1.36%
lat (msec) : 2=13.40%, 4=8.68%, 10=6.46%, 20=3.22%, 50=4.30%
lat (msec) : 100=6.09%, 250=7.50%, 500=3.56%, 750=1.25%, 1000=0.42%
cpu : usr=0.06%, sys=0.27%, ctx=11681, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5582,5765,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=22.1MiB/s (23.2MB/s), 2977KiB/s-19.2MiB/s (3048kB/s-20.1MB/s), io=2664MiB (2793MB), run=120021-120566msec
WRITE: bw=3902KiB/s (3996kB/s), 835KiB/s-3076KiB/s (855kB/s-3150kB/s), io=459MiB (481MB), run=120021-120351msec
Disk stats (read/write):
rbd321: ios=3932/7745, merge=0/1, ticks=559319/910120, in_queue=1463629, util=5.16%
<example-benchmark-fio-server-417-benchmark-gs9tf>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=22, BW=1436KiB/s (1471kB/s)(171MiB/121813msec)
slat (nsec): min=0, max=318460k, avg=653825.73, stdev=10094001.52
clat (nsec): min=0, max=4853.7M, avg=175745024.98, stdev=330156121.69
lat (nsec): min=0, max=4853.0M, avg=176399966.50, stdev=330429601.02
clat percentiles (usec):
| 1.00th=[ 1074], 5.00th=[ 1762], 10.00th=[ 2606],
| 20.00th=[ 4424], 30.00th=[ 7046], 40.00th=[ 10945],
| 50.00th=[ 17433], 60.00th=[ 46400], 70.00th=[ 223347],
| 80.00th=[ 367002], 90.00th=[ 557843], 95.00th=[ 666895],
| 99.00th=[ 943719], 99.50th=[1887437], 99.90th=[4798284],
| 99.95th=[4865393], 99.99th=[4865393]
bw ( KiB/s): min= 0, max= 1607, per=31.79%, avg=1455.50, stdev=214.25, samples=2
iops : min= 0, max= 25, avg=22.50, stdev= 3.54, samples=2
lat (usec) : 4=0.07%, 750=0.07%, 1000=0.66%
lat (msec) : 2=5.42%, 4=12.23%, 10=19.00%, 20=14.54%, 50=8.53%
lat (msec) : 100=4.43%, 250=7.14%, 500=15.49%, 750=9.85%, 1000=1.90%
cpu : usr=0.03%, sys=0.10%, ctx=4795, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2731,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=123, BW=7924KiB/s (8115kB/s)(934MiB/120724msec)
slat (nsec): min=0, max=10609k, avg=31616.62, stdev=166024.40
clat (nsec): min=0, max=1785.3M, avg=32298334.44, stdev=152225712.97
lat (nsec): min=0, max=1785.3M, avg=32330977.46, stdev=152246360.23
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
| 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 844],
| 50.00th=[ 876], 60.00th=[ 908], 70.00th=[ 1112],
| 80.00th=[ 1496], 90.00th=[ 3096576], 95.00th=[ 61603840],
| 99.00th=[ 851443712], 99.50th=[ 977272832], 99.90th=[1266679808],
| 99.95th=[1468006400], 99.99th=[1769996288]
bw ( KiB/s): min= 0, max= 9050, per=72.45%, avg=7959.00, stdev=1542.91, samples=2
iops : min= 0, max= 141, avg=124.00, stdev=24.04, samples=2
lat (nsec) : 750=10.84%, 1000=55.88%
lat (usec) : 2=14.46%, 4=0.16%, 10=0.30%, 20=0.97%, 50=0.21%
lat (usec) : 100=0.13%, 250=0.14%, 500=0.11%, 750=0.27%, 1000=1.05%
lat (msec) : 2=3.48%, 4=3.04%, 10=2.30%, 20=0.89%, 50=0.66%
lat (msec) : 100=0.31%, 250=0.31%, 500=1.02%, 750=1.93%, 1000=1.09%
cpu : usr=0.04%, sys=0.16%, ctx=5094, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=14945,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=48, BW=3080KiB/s (3154kB/s)(361MiB/120004msec)
slat (nsec): min=0, max=6708.7k, avg=29217.79, stdev=128662.97
clat (nsec): min=0, max=1112.9M, avg=9348407.86, stdev=55207623.48
lat (nsec): min=0, max=1112.0M, avg=9378582.70, stdev=55212255.77
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 868], 40.00th=[ 964],
| 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1528],
| 80.00th=[ 148480], 90.00th=[ 2146304], 95.00th=[ 33423360],
| 99.00th=[ 312475648], 99.50th=[ 467664896], 99.90th=[ 624951296],
| 99.95th=[ 742391808], 99.99th=[1115684864]
bw ( KiB/s): min= 0, max= 4738, per=28.03%, avg=3079.50, stdev=2345.47, samples=2
iops : min= 0, max= 74, avg=48.00, stdev=36.77, samples=2
write: IOPS=49, BW=3190KiB/s (3266kB/s)(374MiB/120004msec)
slat (nsec): min=0, max=7569.2k, avg=191882.63, stdev=320415.62
clat (nsec): min=0, max=1050.9M, avg=71244375.78, stdev=134209200.19
lat (nsec): min=0, max=1050.9M, avg=71437577.53, stdev=134155113.65
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1516],
| 20.00th=[ 2073], 30.00th=[ 2900], 40.00th=[ 4359],
| 50.00th=[ 7308], 60.00th=[ 18744], 70.00th=[ 60031],
| 80.00th=[ 109577], 90.00th=[ 217056], 95.00th=[ 379585],
| 99.00th=[ 641729], 99.50th=[ 692061], 99.90th=[ 834667],
| 99.95th=[ 901776], 99.99th=[1052771]
bw ( KiB/s): min= 0, max= 4899, per=69.63%, avg=3187.50, stdev=2420.43, samples=2
iops : min= 0, max= 76, avg=49.50, stdev=37.48, samples=2
lat (nsec) : 750=9.10%, 1000=11.98%
lat (usec) : 2=14.25%, 4=0.09%, 10=0.37%, 20=1.14%, 50=0.95%
lat (usec) : 100=0.88%, 250=1.16%, 500=0.80%, 750=0.48%, 1000=1.41%
lat (msec) : 2=10.99%, 4=10.91%, 10=8.98%, 20=3.69%, 50=4.57%
lat (msec) : 100=5.85%, 250=7.58%, 500=3.06%, 750=1.60%, 1000=0.14%
cpu : usr=0.07%, sys=0.28%, ctx=12640, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5775,5978,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=10.7MiB/s (11.2MB/s), 3080KiB/s-7924KiB/s (3154kB/s-8115kB/s), io=1295MiB (1358MB), run=120004-120724msec
WRITE: bw=4579KiB/s (4689kB/s), 1436KiB/s-3190KiB/s (1471kB/s-3266kB/s), io=545MiB (571MB), run=120004-121813msec
Disk stats (read/write):
rbd128: ios=3825/8987, merge=0/2, ticks=579612/909061, in_queue=1482232, util=3.77%
<example-benchmark-fio-server-702-benchmark-72kl9>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=22, BW=1418KiB/s (1452kB/s)(168MiB/121297msec)
slat (nsec): min=0, max=427537k, avg=661838.85, stdev=12390156.81
clat (nsec): min=0, max=2646.7M, avg=182451904.64, stdev=362745931.13
lat (nsec): min=0, max=2646.8M, avg=183115311.85, stdev=362747429.03
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1549], 10.00th=[ 1893],
| 20.00th=[ 2573], 30.00th=[ 3326], 40.00th=[ 4424],
| 50.00th=[ 6063], 60.00th=[ 9634], 70.00th=[ 23200],
| 80.00th=[ 396362], 90.00th=[ 725615], 95.00th=[ 968885],
| 99.00th=[1535116], 99.50th=[1551893], 99.90th=[2566915],
| 99.95th=[2634023], 99.99th=[2634023]
bw ( KiB/s): min= 0, max= 1689, per=33.81%, avg=1430.50, stdev=365.57, samples=2
iops : min= 0, max= 26, avg=22.00, stdev= 5.66, samples=2
lat (usec) : 750=0.26%, 1000=0.48%
lat (msec) : 2=10.95%, 4=25.33%, 10=23.39%, 20=8.68%, 50=3.87%
lat (msec) : 100=2.38%, 250=2.38%, 500=5.85%, 750=7.56%, 1000=4.17%
cpu : usr=0.03%, sys=0.10%, ctx=4842, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2685,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=202, BW=12.7MiB/s (13.3MB/s)(1525MiB/120525msec)
slat (nsec): min=0, max=67962k, avg=31963.04, stdev=493698.59
clat (nsec): min=0, max=1958.1M, avg=19763723.06, stdev=119137048.25
lat (nsec): min=0, max=1958.2M, avg=19796745.38, stdev=119154585.48
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
| 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
| 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 1004],
| 80.00th=[ 1400], 90.00th=[ 1777664], 95.00th=[ 8028160],
| 99.00th=[ 742391808], 99.50th=[ 893386752], 99.90th=[1249902592],
| 99.95th=[1300234240], 99.99th=[1686110208]
bw ( KiB/s): min= 0, max=15112, per=82.69%, avg=12982.00, stdev=3012.27, samples=2
iops : min= 0, max= 236, avg=202.50, stdev=47.38, samples=2
lat (nsec) : 750=11.73%, 1000=58.10%
lat (usec) : 2=12.66%, 4=0.15%, 10=0.42%, 20=1.68%, 50=0.48%
lat (usec) : 100=0.20%, 250=0.13%, 500=0.11%, 750=0.26%, 1000=1.27%
lat (msec) : 2=3.29%, 4=2.92%, 10=1.97%, 20=0.82%, 50=0.57%
lat (msec) : 100=0.27%, 250=0.28%, 500=0.62%, 750=1.13%, 1000=0.68%
cpu : usr=0.07%, sys=0.24%, ctx=7087, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=24401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=43, BW=2754KiB/s (2820kB/s)(323MiB/120018msec)
slat (nsec): min=0, max=2284.4k, avg=27384.79, stdev=79125.39
clat (nsec): min=0, max=935607k, avg=7895742.05, stdev=47845118.61
lat (nsec): min=0, max=935651k, avg=7924026.04, stdev=47851357.49
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 852], 40.00th=[ 948],
| 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1512],
| 80.00th=[ 126464], 90.00th=[ 2113536], 95.00th=[ 32112640],
| 99.00th=[200278016], 99.50th=[392167424], 99.90th=[641728512],
| 99.95th=[717225984], 99.99th=[935329792]
bw ( KiB/s): min= 0, max= 4245, per=17.54%, avg=2753.50, stdev=2109.30, samples=2
iops : min= 0, max= 66, avg=42.50, stdev=33.23, samples=2
write: IOPS=44, BW=2843KiB/s (2911kB/s)(333MiB/120018msec)
slat (nsec): min=0, max=10903k, avg=179886.99, stdev=309272.59
clat (nsec): min=0, max=1047.2M, avg=82461684.37, stdev=148141789.84
lat (nsec): min=0, max=1047.5M, avg=82642759.28, stdev=148090585.05
clat percentiles (usec):
| 1.00th=[ 971], 5.00th=[ 1205], 10.00th=[ 1385],
| 20.00th=[ 1893], 30.00th=[ 2573], 40.00th=[ 3982],
| 50.00th=[ 9765], 60.00th=[ 26608], 70.00th=[ 79168],
| 80.00th=[ 123208], 90.00th=[ 265290], 95.00th=[ 446694],
| 99.00th=[ 692061], 99.50th=[ 742392], 99.90th=[ 876610],
| 99.95th=[ 910164], 99.99th=[1044382]
bw ( KiB/s): min= 0, max= 4369, per=67.16%, avg=2841.50, stdev=2160.21, samples=2
iops : min= 0, max= 68, avg=44.00, stdev=33.94, samples=2
lat (nsec) : 750=8.04%, 1000=13.42%
lat (usec) : 2=13.92%, 4=0.02%, 10=0.30%, 20=1.21%, 50=0.87%
lat (usec) : 100=1.30%, 250=0.97%, 500=0.61%, 750=0.59%, 1000=1.70%
lat (msec) : 2=12.33%, 4=10.48%, 10=5.81%, 20=3.69%, 50=5.07%
lat (msec) : 100=5.65%, 250=8.36%, 500=3.69%, 750=1.73%, 1000=0.26%
cpu : usr=0.05%, sys=0.25%, ctx=11299, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5164,5328,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=15.3MiB/s (16.1MB/s), 2754KiB/s-12.7MiB/s (2820kB/s-13.3MB/s), io=1848MiB (1938MB), run=120018-120525msec
WRITE: bw=4231KiB/s (4333kB/s), 1418KiB/s-2843KiB/s (1452kB/s-2911kB/s), io=501MiB (526MB), run=120018-121297msec
Disk stats (read/write):
rbd182: ios=4729/8233, merge=0/2, ticks=563379/937246, in_queue=1494170, util=4.62%
<example-benchmark-fio-server-620-benchmark-rlhzh>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=36, BW=2346KiB/s (2402kB/s)(276MiB/120414msec)
slat (nsec): min=0, max=1809.7M, avg=1451143.21, stdev=41416492.97
clat (nsec): min=0, max=2608.3M, avg=107987024.10, stdev=273072237.75
lat (nsec): min=0, max=2673.9M, avg=109505679.21, stdev=278381371.46
clat percentiles (usec):
| 1.00th=[ 1139], 5.00th=[ 1450], 10.00th=[ 1663],
| 20.00th=[ 2147], 30.00th=[ 2769], 40.00th=[ 3654],
| 50.00th=[ 5145], 60.00th=[ 8586], 70.00th=[ 16909],
| 80.00th=[ 50594], 90.00th=[ 488637], 95.00th=[ 759170],
| 99.00th=[1283458], 99.50th=[1384121], 99.90th=[1870660],
| 99.95th=[1887437], 99.99th=[2600469]
bw ( KiB/s): min= 0, max= 3078, per=46.17%, avg=2351.50, stdev=1027.43, samples=2
iops : min= 0, max= 48, avg=36.50, stdev=16.26, samples=2
lat (usec) : 500=0.02%, 750=0.09%, 1000=0.25%
lat (msec) : 2=16.78%, 4=26.14%, 10=19.43%, 20=9.00%, 50=8.34%
lat (msec) : 100=5.01%, 250=2.68%, 500=2.40%, 750=4.87%, 1000=2.47%
cpu : usr=0.04%, sys=0.17%, ctx=8054, majf=0, minf=29
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4411,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=832, BW=51.8MiB/s (54.4MB/s)(6223MiB/120042msec)
slat (nsec): min=0, max=1627.4M, avg=61070.51, stdev=7625575.00
clat (nsec): min=0, max=2303.2M, avg=4747375.68, stdev=57025931.76
lat (nsec): min=0, max=2303.2M, avg=4809469.51, stdev=57538826.11
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
| 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
| 80.00th=[ 1400], 90.00th=[ 12864], 95.00th=[ 782336],
| 99.00th=[ 43778048], 99.50th=[ 438304768], 99.90th=[ 901775360],
| 99.95th=[1027604480], 99.99th=[1417674752]
bw ( KiB/s): min= 0, max=88652, per=95.26%, avg=53098.00, stdev=50280.95, samples=2
iops : min= 0, max= 1391, avg=832.50, stdev=789.84, samples=2
lat (nsec) : 750=14.75%, 1000=60.85%
lat (usec) : 2=5.43%, 4=0.15%, 10=1.52%, 20=9.01%, 50=2.48%
lat (usec) : 100=0.27%, 250=0.18%, 500=0.12%, 750=0.17%, 1000=0.56%
lat (msec) : 2=1.15%, 4=0.85%, 10=0.60%, 20=0.40%, 50=0.61%
lat (msec) : 100=0.20%, 250=0.10%, 500=0.16%, 750=0.25%, 1000=0.14%
cpu : usr=0.22%, sys=0.80%, ctx=11085, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=99919,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=41, BW=2655KiB/s (2719kB/s)(311MiB/120015msec)
slat (nsec): min=0, max=4456.3k, avg=32168.09, stdev=97756.59
clat (nsec): min=0, max=989312k, avg=12693057.98, stdev=60321579.79
lat (nsec): min=0, max=989349k, avg=12726223.12, stdev=60329190.75
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
| 20.00th=[ 820], 30.00th=[ 916], 40.00th=[ 1080],
| 50.00th=[ 1224], 60.00th=[ 1384], 70.00th=[ 44288],
| 80.00th=[ 1122304], 90.00th=[ 15138816], 95.00th=[ 62128128],
| 99.00th=[320864256], 99.50th=[425721856], 99.90th=[767557632],
| 99.95th=[935329792], 99.99th=[985661440]
bw ( KiB/s): min= 0, max= 3947, per=4.76%, avg=2655.00, stdev=1827.16, samples=2
iops : min= 0, max= 61, avg=41.00, stdev=28.28, samples=2
write: IOPS=43, BW=2757KiB/s (2823kB/s)(323MiB/120015msec)
slat (nsec): min=0, max=8114.2k, avg=169031.29, stdev=274467.99
clat (nsec): min=0, max=1041.7M, avg=80800586.43, stdev=146966844.61
lat (nsec): min=0, max=1041.8M, avg=80970787.08, stdev=146927156.52
clat percentiles (usec):
| 1.00th=[ 1057], 5.00th=[ 1270], 10.00th=[ 1483],
| 20.00th=[ 1926], 30.00th=[ 2671], 40.00th=[ 4047],
| 50.00th=[ 8586], 60.00th=[ 24773], 70.00th=[ 65799],
| 80.00th=[ 111674], 90.00th=[ 295699], 95.00th=[ 455082],
| 99.00th=[ 641729], 99.50th=[ 700449], 99.90th=[ 851444],
| 99.95th=[ 868221], 99.99th=[1044382]
bw ( KiB/s): min= 0, max= 4086, per=54.10%, avg=2755.50, stdev=1881.61, samples=2
iops : min= 0, max= 63, avg=42.50, stdev=28.99, samples=2
lat (nsec) : 750=6.03%, 1000=11.10%
lat (usec) : 2=15.61%, 4=0.11%, 10=0.19%, 20=0.82%, 50=0.69%
lat (usec) : 100=0.59%, 250=0.89%, 500=0.49%, 750=0.73%, 1000=1.93%
lat (msec) : 2=12.85%, 4=11.24%, 10=6.76%, 20=3.80%, 50=6.10%
lat (msec) : 100=6.97%, 250=6.51%, 500=4.52%, 750=1.90%, 1000=0.18%
cpu : usr=0.07%, sys=0.23%, ctx=12113, majf=0, minf=33
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4978,5168,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=54.4MiB/s (57.1MB/s), 2655KiB/s-51.8MiB/s (2719kB/s-54.4MB/s), io=6534MiB (6852MB), run=120015-120042msec
WRITE: bw=5094KiB/s (5216kB/s), 2346KiB/s-2757KiB/s (2402kB/s-2823kB/s), io=599MiB (628MB), run=120015-120414msec
Disk stats (read/write):
rbd291: ios=6612/9793, merge=0/1, ticks=541800/872212, in_queue=1405759, util=7.93%
<example-benchmark-fio-server-139-benchmark-7btww>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=21, BW=1355KiB/s (1388kB/s)(160MiB/120980msec)
slat (nsec): min=0, max=859342k, avg=1241964.81, stdev=25538249.96
clat (nsec): min=0, max=2242.1M, avg=188335014.13, stdev=370328253.41
lat (nsec): min=0, max=2242.8M, avg=189577480.82, stdev=371271934.52
clat percentiles (usec):
| 1.00th=[ 1090], 5.00th=[ 1434], 10.00th=[ 1778],
| 20.00th=[ 2474], 30.00th=[ 3326], 40.00th=[ 4752],
| 50.00th=[ 7242], 60.00th=[ 13566], 70.00th=[ 27657],
| 80.00th=[ 392168], 90.00th=[ 817890], 95.00th=[1035994],
| 99.00th=[1484784], 99.50th=[1635779], 99.90th=[2139096],
| 99.95th=[2139096], 99.99th=[2231370]
bw ( KiB/s): min= 0, max= 1415, per=29.91%, avg=1358.50, stdev=79.90, samples=2
iops : min= 0, max= 22, avg=21.00, stdev= 1.41, samples=2
lat (usec) : 10=0.04%, 500=0.04%, 750=0.08%, 1000=0.31%
lat (msec) : 2=12.90%, 4=21.81%, 10=20.75%, 20=9.73%, 50=8.75%
lat (msec) : 100=1.99%, 250=1.80%, 500=4.85%, 750=5.31%, 1000=5.90%
cpu : usr=0.02%, sys=0.11%, ctx=4636, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2559,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=376, BW=23.5MiB/s (24.7MB/s)(2837MiB/120564msec)
slat (nsec): min=0, max=691432k, avg=43024.80, stdev=3613085.44
clat (nsec): min=0, max=2143.1M, avg=10554204.56, stdev=88854248.08
lat (nsec): min=0, max=2143.2M, avg=10598187.22, stdev=88937193.66
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
| 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
| 50.00th=[ 860], 60.00th=[ 892], 70.00th=[ 932],
| 80.00th=[ 1144], 90.00th=[ 13632], 95.00th=[ 2342912],
| 99.00th=[ 541065216], 99.50th=[ 759169024], 99.90th=[1052770304],
| 99.95th=[1266679808], 99.99th=[1602224128]
bw ( KiB/s): min= 0, max=34499, per=89.06%, avg=24203.50, stdev=14560.04, samples=2
iops : min= 0, max= 539, avg=378.00, stdev=227.69, samples=2
lat (nsec) : 750=10.71%, 1000=65.67%
lat (usec) : 2=10.08%, 4=0.17%, 10=0.54%, 20=3.27%, 50=1.05%
lat (usec) : 100=0.19%, 250=0.11%, 500=0.19%, 750=0.19%, 1000=0.72%
lat (msec) : 2=1.76%, 4=1.56%, 10=1.17%, 20=0.52%, 50=0.42%
lat (msec) : 100=0.13%, 250=0.16%, 500=0.33%, 750=0.55%, 1000=0.37%
cpu : usr=0.10%, sys=0.42%, ctx=7607, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=45440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=48, BW=3100KiB/s (3175kB/s)(363MiB/120004msec)
slat (nsec): min=0, max=3502.5k, avg=30685.11, stdev=91394.21
clat (nsec): min=0, max=886323k, avg=8285032.64, stdev=51086846.62
lat (nsec): min=0, max=886388k, avg=8316623.23, stdev=51091851.38
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 996],
| 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1672],
| 80.00th=[ 179200], 90.00th=[ 1974272], 95.00th=[ 14221312],
| 99.00th=[320864256], 99.50th=[438304768], 99.90th=[583008256],
| 99.95th=[658505728], 99.99th=[884998144]
bw ( KiB/s): min= 0, max= 4320, per=11.41%, avg=3100.00, stdev=1725.34, samples=2
iops : min= 0, max= 67, avg=48.00, stdev=26.87, samples=2
write: IOPS=50, BW=3213KiB/s (3290kB/s)(377MiB/120004msec)
slat (nsec): min=0, max=13105k, avg=189168.32, stdev=269636.09
clat (nsec): min=0, max=1091.8M, avg=71891687.37, stdev=135619142.37
lat (nsec): min=0, max=1091.9M, avg=72082105.36, stdev=135568803.97
clat percentiles (usec):
| 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1418],
| 20.00th=[ 1827], 30.00th=[ 2442], 40.00th=[ 3818],
| 50.00th=[ 7832], 60.00th=[ 21103], 70.00th=[ 68682],
| 80.00th=[ 107480], 90.00th=[ 200279], 95.00th=[ 392168],
| 99.00th=[ 658506], 99.50th=[ 717226], 99.90th=[ 859833],
| 99.95th=[ 926942], 99.99th=[1098908]
bw ( KiB/s): min= 0, max= 4449, per=70.71%, avg=3211.50, stdev=1750.09, samples=2
iops : min= 0, max= 69, avg=49.50, stdev=27.58, samples=2
lat (nsec) : 750=8.45%, 1000=11.48%
lat (usec) : 2=14.69%, 4=0.07%, 10=0.48%, 20=1.30%, 50=0.89%
lat (usec) : 100=1.14%, 250=1.15%, 500=0.68%, 750=0.79%, 1000=1.64%
lat (msec) : 2=13.27%, 4=10.32%, 10=6.92%, 20=3.79%, 50=4.01%
lat (msec) : 100=6.76%, 250=7.48%, 500=3.02%, 750=1.48%, 1000=0.19%
cpu : usr=0.08%, sys=0.28%, ctx=12857, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5813,6022,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=26.5MiB/s (27.8MB/s), 3100KiB/s-23.5MiB/s (3175kB/s-24.7MB/s), io=3200MiB (3355MB), run=120004-120564msec
WRITE: bw=4543KiB/s (4652kB/s), 1355KiB/s-3213KiB/s (1388kB/s-3290kB/s), io=537MiB (563MB), run=120004-120980msec
Disk stats (read/write):
rbd58: ios=5113/8988, merge=0/1, ticks=567154/914911, in_queue=1474958, util=6.12%
<example-benchmark-fio-server-289-benchmark-jhcgn>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
write: IOPS=44, BW=2824KiB/s (2892kB/s)(331MiB/120072msec)
slat (nsec): min=0, max=714127k, avg=491923.63, stdev=13745623.96
clat (nsec): min=0, max=1605.9M, avg=90397146.42, stdev=161216053.44
lat (nsec): min=0, max=1766.4M, avg=90890178.16, stdev=162453244.77
clat percentiles (usec):
| 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1270],
| 20.00th=[ 1614], 30.00th=[ 2114], 40.00th=[ 3064],
| 50.00th=[ 7046], 60.00th=[ 62129], 70.00th=[ 94897],
| 80.00th=[ 135267], 90.00th=[ 254804], 95.00th=[ 438305],
| 99.00th=[ 759170], 99.50th=[ 943719], 99.90th=[1132463],
| 99.95th=[1149240], 99.99th=[1602225]
bw ( KiB/s): min= 0, max= 3586, per=45.15%, avg=2823.00, stdev=1079.04, samples=2
iops : min= 0, max= 56, avg=44.00, stdev=16.97, samples=2
lat (usec) : 250=0.02%, 500=0.02%, 750=0.09%, 1000=2.00%
lat (msec) : 2=26.38%, 4=15.98%, 10=7.84%, 20=2.72%, 50=3.36%
lat (msec) : 100=13.54%, 250=17.79%, 500=6.35%, 750=2.85%, 1000=0.64%
cpu : usr=0.07%, sys=0.18%, ctx=9958, majf=0, minf=26
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5295,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
read: IOPS=311, BW=19.5MiB/s (20.4MB/s)(2338MiB/120002msec)
slat (nsec): min=0, max=710630k, avg=63218.86, stdev=4979859.96
clat (nsec): min=0, max=1434.2M, avg=12791436.07, stdev=62473523.58
lat (nsec): min=0, max=1434.4M, avg=12855512.68, stdev=62676595.96
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
| 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 1032],
| 80.00th=[ 8768], 90.00th=[ 7045120], 95.00th=[ 60030976],
| 99.00th=[ 362807296], 99.50th=[ 484442112], 99.90th=[ 683671552],
| 99.95th=[ 767557632], 99.99th=[1019215872]
bw ( KiB/s): min= 0, max=25118, per=85.72%, avg=19949.00, stdev=7310.07, samples=2
iops : min= 0, max= 392, avg=311.00, stdev=114.55, samples=2
lat (nsec) : 750=41.73%, 1000=27.46%
lat (usec) : 2=10.55%, 4=0.03%, 10=0.58%, 20=0.31%, 50=0.27%
lat (usec) : 100=0.09%, 250=0.07%, 500=0.27%, 750=0.98%, 1000=2.19%
lat (msec) : 2=3.06%, 4=1.82%, 10=0.83%, 20=0.52%, 50=3.20%
lat (msec) : 100=2.93%, 250=1.53%, 500=1.11%, 750=0.40%, 1000=0.05%
cpu : usr=0.11%, sys=0.35%, ctx=14140, majf=0, minf=97
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=37428,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
read: IOPS=51, BW=3322KiB/s (3402kB/s)(389MiB/120007msec)
slat (nsec): min=0, max=663988k, avg=171583.65, stdev=8737229.14
clat (nsec): min=0, max=632295k, avg=6775755.36, stdev=37927924.18
lat (nsec): min=0, max=664065k, avg=6948327.42, stdev=38902864.79
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
| 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1004],
| 50.00th=[ 1160], 60.00th=[ 1304], 70.00th=[ 17792],
| 80.00th=[ 864256], 90.00th=[ 3031040], 95.00th=[ 33816576],
| 99.00th=[179306496], 99.50th=[308281344], 99.90th=[492830720],
| 99.95th=[574619648], 99.99th=[633339904]
bw ( KiB/s): min= 0, max= 4763, per=14.27%, avg=3322.00, stdev=2037.88, samples=2
iops : min= 0, max= 74, avg=51.50, stdev=31.82, samples=2
write: IOPS=53, BW=3430KiB/s (3513kB/s)(402MiB/120007msec)
slat (nsec): min=0, max=11878k, avg=197961.98, stdev=307898.64
clat (nsec): min=0, max=1469.7M, avg=68202610.29, stdev=138950257.20
lat (nsec): min=0, max=1469.8M, avg=68401706.46, stdev=138903708.14
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1401],
| 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 2999],
| 50.00th=[ 4621], 60.00th=[ 12387], 70.00th=[ 57410],
| 80.00th=[ 103285], 90.00th=[ 200279], 95.00th=[ 354419],
| 99.00th=[ 683672], 99.50th=[ 801113], 99.90th=[1098908],
| 99.95th=[1132463], 99.99th=[1468007]
bw ( KiB/s): min= 0, max= 4867, per=54.84%, avg=3428.50, stdev=2034.35, samples=2
iops : min= 0, max= 76, avg=53.50, stdev=31.82, samples=2
lat (nsec) : 750=7.08%, 1000=12.30%
lat (usec) : 2=14.32%, 4=0.03%, 10=0.24%, 20=0.81%, 50=0.70%
lat (usec) : 100=0.59%, 250=0.86%, 500=0.89%, 750=0.79%, 1000=2.20%
lat (msec) : 2=15.49%, 4=12.24%, 10=7.09%, 20=2.65%, 50=4.11%
lat (msec) : 100=6.31%, 250=7.16%, 500=2.66%, 750=1.15%, 1000=0.28%
cpu : usr=0.07%, sys=0.31%, ctx=14996, majf=0, minf=33
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6231,6429,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=22.7MiB/s (23.8MB/s), 3322KiB/s-19.5MiB/s (3402kB/s-20.4MB/s), io=2728MiB (2860MB), run=120002-120007msec
WRITE: bw=6252KiB/s (6402kB/s), 2824KiB/s-3430KiB/s (2892kB/s-3513kB/s), io=733MiB (769MB), run=120007-120072msec
Disk stats (read/write):
rbd123: ios=9289/12253, merge=0/2, ticks=557195/852287, in_queue=1398563, util=14.81%
<example-benchmark-fio-server-115-benchmark-krrp7>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=26, BW=1687KiB/s (1727kB/s)(198MiB/120401msec)
slat (nsec): min=0, max=464962k, avg=535888.93, stdev=11242241.37
clat (nsec): min=0, max=2030.7M, avg=151464747.66, stdev=258316057.55
lat (nsec): min=0, max=2030.8M, avg=152001879.93, stdev=258431932.77
clat percentiles (usec):
| 1.00th=[ 1270], 5.00th=[ 1942], 10.00th=[ 2573],
| 20.00th=[ 3851], 30.00th=[ 5473], 40.00th=[ 7832],
| 50.00th=[ 11207], 60.00th=[ 20055], 70.00th=[ 122160],
| 80.00th=[ 329253], 90.00th=[ 549454], 95.00th=[ 700449],
| 99.00th=[ 943719], 99.50th=[1182794], 99.90th=[2021655],
| 99.95th=[2021655], 99.99th=[2038432]
bw ( KiB/s): min= 0, max= 1993, per=31.33%, avg=1688.50, stdev=430.63, samples=2
iops : min= 0, max= 31, avg=26.00, stdev= 7.07, samples=2
lat (usec) : 4=0.13%, 250=0.03%, 500=0.03%, 750=0.06%, 1000=0.06%
lat (msec) : 2=5.21%, 4=15.36%, 10=26.12%, 20=13.19%, 50=6.50%
lat (msec) : 100=2.62%, 250=6.18%, 500=13.15%, 750=7.48%, 1000=3.19%
cpu : usr=0.03%, sys=0.12%, ctx=5496, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3170,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=123, BW=7920KiB/s (8110kB/s)(933MiB/120639msec)
slat (nsec): min=0, max=7129.5k, avg=37269.24, stdev=139259.60
clat (nsec): min=0, max=1959.1M, avg=32268522.52, stdev=150613641.83
lat (nsec): min=0, max=1959.1M, avg=32306809.86, stdev=150630782.48
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
| 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
| 50.00th=[ 908], 60.00th=[ 1080], 70.00th=[ 1368],
| 80.00th=[ 1138688], 90.00th=[ 4882432], 95.00th=[ 56885248],
| 99.00th=[ 843055104], 99.50th=[ 960495616], 99.90th=[1216348160],
| 99.95th=[1400897536], 99.99th=[1803550720]
bw ( KiB/s): min= 0, max= 8521, per=68.97%, avg=7955.50, stdev=799.74, samples=2
iops : min= 0, max= 133, avg=124.00, stdev=12.73, samples=2
lat (nsec) : 750=10.00%, 1000=47.62%
lat (usec) : 2=17.35%, 4=0.25%, 10=0.15%, 20=0.74%, 50=0.16%
lat (usec) : 100=0.14%, 250=0.13%, 500=0.11%, 750=0.44%, 1000=1.89%
lat (msec) : 2=5.43%, 4=4.56%, 10=3.44%, 20=1.44%, 50=1.05%
lat (msec) : 100=0.33%, 250=0.31%, 500=1.02%, 750=1.94%, 1000=1.11%
cpu : usr=0.04%, sys=0.18%, ctx=7043, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=14926,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=56, BW=3635KiB/s (3722kB/s)(426MiB/120006msec)
slat (nsec): min=0, max=1680.6k, avg=30012.19, stdev=72067.62
clat (nsec): min=0, max=747366k, avg=10032263.04, stdev=54279230.59
lat (nsec): min=0, max=747386k, avg=10063205.13, stdev=54284115.55
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1012],
| 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 15808],
| 80.00th=[ 921600], 90.00th=[ 3522560], 95.00th=[ 45350912],
| 99.00th=[329252864], 99.50th=[442499072], 99.90th=[675282944],
| 99.95th=[700448768], 99.99th=[750780416]
bw ( KiB/s): min= 0, max= 5302, per=31.51%, avg=3635.00, stdev=2357.49, samples=2
iops : min= 0, max= 82, avg=56.00, stdev=36.77, samples=2
write: IOPS=58, BW=3715KiB/s (3804kB/s)(435MiB/120006msec)
slat (nsec): min=0, max=29166k, avg=196391.82, stdev=440398.02
clat (nsec): min=0, max=905267k, avg=58898123.51, stdev=124023895.80
lat (nsec): min=0, max=906784k, avg=59095996.36, stdev=123976481.96
clat percentiles (usec):
| 1.00th=[ 979], 5.00th=[ 1172], 10.00th=[ 1352], 20.00th=[ 1680],
| 30.00th=[ 2114], 40.00th=[ 2868], 50.00th=[ 4686], 60.00th=[ 10421],
| 70.00th=[ 32113], 80.00th=[ 93848], 90.00th=[160433], 95.00th=[320865],
| 99.00th=[675283], 99.50th=[767558], 99.90th=[868221], 99.95th=[893387],
| 99.99th=[901776]
bw ( KiB/s): min= 0, max= 5360, per=68.90%, avg=3713.00, stdev=2329.21, samples=2
iops : min= 0, max= 83, avg=57.50, stdev=36.06, samples=2
lat (nsec) : 750=7.63%, 1000=11.81%
lat (usec) : 2=14.57%, 4=0.04%, 10=0.25%, 20=0.73%, 50=0.65%
lat (usec) : 100=0.67%, 250=0.86%, 500=0.48%, 750=0.68%, 1000=2.32%
lat (msec) : 2=16.48%, 4=11.47%, 10=6.95%, 20=4.01%, 50=4.48%
lat (msec) : 100=5.18%, 250=6.82%, 500=2.74%, 750=0.91%, 1000=0.28%
cpu : usr=0.08%, sys=0.33%, ctx=15897, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6816,6963,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=11.3MiB/s (11.8MB/s), 3635KiB/s-7920KiB/s (3722kB/s-8110kB/s), io=1359MiB (1425MB), run=120006-120639msec
WRITE: bw=5389KiB/s (5519kB/s), 1687KiB/s-3715KiB/s (1727kB/s-3804kB/s), io=634MiB (664MB), run=120006-120401msec
Disk stats (read/write):
rbd33: ios=5721/11388, merge=0/3, ticks=605434/883833, in_queue=1480636, util=5.12%
<example-benchmark-fio-server-594-benchmark-dz8fd>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=39, BW=2556KiB/s (2617kB/s)(300MiB/120181msec)
slat (nsec): min=0, max=746936k, avg=484272.82, stdev=14806221.57
clat (nsec): min=0, max=1378.3M, avg=100241276.02, stdev=175257826.40
lat (nsec): min=0, max=1378.4M, avg=100726680.74, stdev=175669845.78
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1450],
| 20.00th=[ 1926], 30.00th=[ 2868], 40.00th=[ 4883],
| 50.00th=[ 11076], 60.00th=[ 55837], 70.00th=[ 99091],
| 80.00th=[ 139461], 90.00th=[ 337642], 95.00th=[ 484443],
| 99.00th=[ 809501], 99.50th=[ 952108], 99.90th=[1283458],
| 99.95th=[1317012], 99.99th=[1384121]
bw ( KiB/s): min= 0, max= 2893, per=46.14%, avg=2557.00, stdev=475.18, samples=2
iops : min= 0, max= 45, avg=39.50, stdev= 7.78, samples=2
lat (usec) : 100=0.02%, 500=0.02%, 750=0.04%, 1000=0.92%
lat (msec) : 2=20.20%, 4=15.22%, 10=12.42%, 20=5.88%, 50=4.48%
lat (msec) : 100=11.03%, 250=16.76%, 500=8.46%, 750=3.25%, 1000=0.96%
cpu : usr=0.05%, sys=0.17%, ctx=8988, majf=0, minf=30
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4797,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2525MiB/120182msec)
slat (nsec): min=0, max=688109k, avg=55153.02, stdev=4613265.28
clat (nsec): min=0, max=1316.2M, avg=11849012.02, stdev=62320874.61
lat (nsec): min=0, max=1316.2M, avg=11905146.43, stdev=62497607.01
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
| 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1112],
| 80.00th=[ 354304], 90.00th=[ 2998272], 95.00th=[ 55312384],
| 99.00th=[ 358612992], 99.50th=[ 471859200], 99.90th=[ 725614592],
| 99.95th=[ 859832320], 99.99th=[1199570944]
bw ( KiB/s): min= 0, max=31963, per=88.33%, avg=21542.00, stdev=14737.52, samples=2
iops : min= 0, max= 499, avg=336.00, stdev=230.52, samples=2
lat (nsec) : 750=38.94%, 1000=27.33%
lat (usec) : 2=11.77%, 4=0.04%, 10=0.52%, 20=0.59%, 50=0.19%
lat (usec) : 100=0.11%, 250=0.08%, 500=0.87%, 750=1.40%, 1000=2.68%
lat (msec) : 2=3.81%, 4=2.41%, 10=1.09%, 20=0.57%, 50=2.13%
lat (msec) : 100=2.62%, 250=1.33%, 500=1.11%, 750=0.33%, 1000=0.07%
cpu : usr=0.11%, sys=0.40%, ctx=16448, majf=0, minf=108
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40399,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=45, BW=2881KiB/s (2950kB/s)(338MiB/120007msec)
slat (nsec): min=0, max=3273.4k, avg=38406.71, stdev=105301.68
clat (nsec): min=0, max=585036k, avg=9336840.69, stdev=43652742.88
lat (nsec): min=0, max=585053k, avg=9376204.46, stdev=43661535.41
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
| 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1064],
| 50.00th=[ 1208], 60.00th=[ 1400], 70.00th=[ 209920],
| 80.00th=[ 1155072], 90.00th=[ 6782976], 95.00th=[ 46399488],
| 99.00th=[244318208], 99.50th=[350224384], 99.90th=[509607936],
| 99.95th=[541065216], 99.99th=[583008256]
bw ( KiB/s): min= 0, max= 4260, per=11.81%, avg=2879.50, stdev=1952.32, samples=2
iops : min= 0, max= 66, avg=44.50, stdev=30.41, samples=2
write: IOPS=46, BW=2991KiB/s (3063kB/s)(351MiB/120007msec)
slat (nsec): min=0, max=57340k, avg=188996.30, stdev=811463.73
clat (nsec): min=0, max=1109.1M, avg=76748413.93, stdev=143499222.92
lat (nsec): min=0, max=1109.1M, avg=76938515.68, stdev=143449607.79
clat percentiles (usec):
| 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1532],
| 20.00th=[ 2024], 30.00th=[ 2900], 40.00th=[ 4817],
| 50.00th=[ 10814], 60.00th=[ 26346], 70.00th=[ 67634],
| 80.00th=[ 107480], 90.00th=[ 235930], 95.00th=[ 425722],
| 99.00th=[ 692061], 99.50th=[ 759170], 99.90th=[ 943719],
| 99.95th=[ 985662], 99.99th=[1115685]
bw ( KiB/s): min= 0, max= 4384, per=53.94%, avg=2989.50, stdev=1972.12, samples=2
iops : min= 0, max= 68, avg=46.00, stdev=31.11, samples=2
lat (nsec) : 750=6.66%, 1000=10.83%
lat (usec) : 2=14.09%, 4=0.10%, 10=0.29%, 20=0.60%, 50=0.65%
lat (usec) : 100=0.66%, 250=0.60%, 500=1.10%, 750=1.01%, 1000=2.00%
lat (msec) : 2=12.96%, 4=10.63%, 10=7.37%, 20=4.72%, 50=6.18%
lat (msec) : 100=7.20%, 250=7.00%, 500=3.70%, 750=1.34%, 1000=0.31%
cpu : usr=0.06%, sys=0.28%, ctx=13512, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5402,5605,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=23.8MiB/s (24.0MB/s), 2881KiB/s-21.0MiB/s (2950kB/s-22.0MB/s), io=2862MiB (3001MB), run=120007-120182msec
WRITE: bw=5543KiB/s (5676kB/s), 2556KiB/s-2991KiB/s (2617kB/s-3063kB/s), io=651MiB (682MB), run=120007-120181msec
Disk stats (read/write):
rbd248: ios=11875/11980, merge=0/2, ticks=556245/857749, in_queue=1401822, util=15.70%
<example-benchmark-fio-server-130-benchmark-ckmtx>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=51, BW=3328KiB/s (3408kB/s)(390MiB/120061msec)
slat (nsec): min=0, max=657788k, avg=347116.30, stdev=8981861.13
clat (nsec): min=0, max=1140.8M, avg=76624588.97, stdev=145436001.13
lat (nsec): min=0, max=1140.8M, avg=76973399.87, stdev=145639513.75
clat percentiles (usec):
| 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1319],
| 20.00th=[ 1696], 30.00th=[ 2376], 40.00th=[ 3589],
| 50.00th=[ 7177], 60.00th=[ 24773], 70.00th=[ 71828],
| 80.00th=[ 108528], 90.00th=[ 233833], 95.00th=[ 400557],
| 99.00th=[ 725615], 99.50th=[ 843056], 99.90th=[ 952108],
| 99.95th=[1002439], 99.99th=[1132463]
bw ( KiB/s): min= 0, max= 3477, per=50.05%, avg=3327.50, stdev=211.42, samples=2
iops : min= 0, max= 54, avg=51.50, stdev= 3.54, samples=2
lat (usec) : 2=0.02%, 4=0.02%, 500=0.03%, 750=0.05%, 1000=1.47%
lat (msec) : 2=23.52%, 4=17.03%, 10=11.04%, 20=5.26%, 50=6.68%
lat (msec) : 100=12.39%, 250=13.27%, 500=6.18%, 750=2.20%, 1000=0.87%
cpu : usr=0.07%, sys=0.22%, ctx=11790, majf=0, minf=26
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6241,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=338, BW=21.1MiB/s (22.1MB/s)(2533MiB/120041msec)
slat (nsec): min=0, max=520529k, avg=41330.26, stdev=2721988.27
clat (nsec): min=0, max=1107.1M, avg=11867312.79, stdev=59671897.16
lat (nsec): min=0, max=1107.1M, avg=11909494.08, stdev=59741193.49
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
| 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
| 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
| 80.00th=[ 24960], 90.00th=[ 5210112], 95.00th=[ 56360960],
| 99.00th=[ 337641472], 99.50th=[ 471859200], 99.90th=[ 700448768],
| 99.95th=[ 759169024], 99.99th=[1061158912]
bw ( KiB/s): min= 0, max=31047, per=87.08%, avg=21615.00, stdev=13338.86, samples=2
iops : min= 0, max= 485, avg=337.50, stdev=208.60, samples=2
lat (nsec) : 750=41.36%, 1000=26.06%
lat (usec) : 2=10.59%, 4=0.04%, 10=0.87%, 20=1.02%, 50=0.41%
lat (usec) : 100=0.08%, 250=0.04%, 500=0.19%, 750=1.10%, 1000=2.41%
lat (msec) : 2=3.42%, 4=2.02%, 10=0.98%, 20=0.62%, 50=3.06%
lat (msec) : 100=2.91%, 250=1.43%, 500=0.96%, 750=0.36%, 1000=0.05%
cpu : usr=0.11%, sys=0.39%, ctx=15868, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=40576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=50, BW=3214KiB/s (3291kB/s)(377MiB/120006msec)
slat (nsec): min=0, max=657744k, avg=145579.23, stdev=8474050.87
clat (nsec): min=0, max=1066.6M, avg=8382053.59, stdev=40166155.39
lat (nsec): min=0, max=1066.6M, avg=8528658.07, stdev=41033365.30
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
| 20.00th=[ 804], 30.00th=[ 908], 40.00th=[ 1048],
| 50.00th=[ 1176], 60.00th=[ 1320], 70.00th=[ 18560],
| 80.00th=[ 1003520], 90.00th=[ 7897088], 95.00th=[ 48496640],
| 99.00th=[ 183500800], 99.50th=[ 274726912], 99.90th=[ 484442112],
| 99.95th=[ 517996544], 99.99th=[1069547520]
bw ( KiB/s): min= 0, max= 4074, per=12.95%, avg=3214.00, stdev=1216.22, samples=2
iops : min= 0, max= 63, avg=49.50, stdev=19.09, samples=2
write: IOPS=51, BW=3323KiB/s (3403kB/s)(389MiB/120006msec)
slat (nsec): min=0, max=63599k, avg=206804.07, stdev=945686.55
clat (nsec): min=0, max=1233.8M, avg=68800246.38, stdev=134987032.75
lat (nsec): min=0, max=1233.9M, avg=69008286.97, stdev=134933795.73
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1434],
| 20.00th=[ 1942], 30.00th=[ 2769], 40.00th=[ 4490],
| 50.00th=[ 8717], 60.00th=[ 21890], 70.00th=[ 54264],
| 80.00th=[ 96994], 90.00th=[ 204473], 95.00th=[ 375391],
| 99.00th=[ 650118], 99.50th=[ 775947], 99.90th=[ 943719],
| 99.95th=[ 952108], 99.99th=[1233126]
bw ( KiB/s): min= 0, max= 4184, per=49.95%, avg=3321.00, stdev=1220.47, samples=2
iops : min= 0, max= 65, avg=51.50, stdev=19.09, samples=2
lat (nsec) : 750=6.62%, 1000=11.64%
lat (usec) : 2=15.40%, 4=0.07%, 10=0.26%, 20=0.58%, 50=0.62%
lat (usec) : 100=0.55%, 250=0.57%, 500=0.71%, 750=0.72%, 1000=2.13%
lat (msec) : 2=12.83%, 4=10.11%, 10=7.98%, 20=4.37%, 50=6.58%
lat (msec) : 100=7.44%, 250=6.36%, 500=3.20%, 750=0.96%, 1000=0.32%
cpu : usr=0.09%, sys=0.29%, ctx=14868, majf=0, minf=34
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6028,6228,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=24.2MiB/s (25.4MB/s), 3214KiB/s-21.1MiB/s (3291kB/s-22.1MB/s), io=2910MiB (3051MB), run=120006-120041msec
WRITE: bw=6650KiB/s (6810kB/s), 3323KiB/s-3328KiB/s (3403kB/s-3408kB/s), io=780MiB (818MB), run=120006-120061msec
Disk stats (read/write):
rbd49: ios=9506/12982, merge=0/1, ticks=545666/834922, in_queue=1369109, util=15.51%
<example-benchmark-fio-server-600-benchmark-fvxhx>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=19, BW=1236KiB/s (1265kB/s)(145MiB/120516msec)
slat (nsec): min=0, max=694928k, avg=1293864.89, stdev=25365740.43
clat (nsec): min=0, max=2243.2M, avg=206617078.50, stdev=402630615.34
lat (nsec): min=0, max=2243.3M, avg=207910976.65, stdev=403420903.13
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1647],
| 20.00th=[ 2180], 30.00th=[ 2802], 40.00th=[ 3556],
| 50.00th=[ 4817], 60.00th=[ 7832], 70.00th=[ 21103],
| 80.00th=[ 467665], 90.00th=[ 884999], 95.00th=[1199571],
| 99.00th=[1501561], 99.50th=[1635779], 99.90th=[1920992],
| 99.95th=[2139096], 99.99th=[2231370]
bw ( KiB/s): min= 0, max= 1654, per=26.83%, avg=1238.50, stdev=587.61, samples=2
iops : min= 0, max= 25, avg=18.50, stdev= 9.19, samples=2
lat (usec) : 750=0.09%, 1000=0.17%
lat (msec) : 2=16.44%, 4=27.54%, 10=18.76%, 20=6.45%, 50=5.16%
lat (msec) : 100=1.42%, 250=1.98%, 500=2.41%, 750=6.88%, 1000=5.59%
cpu : usr=0.02%, sys=0.09%, ctx=4282, majf=0, minf=29
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2324,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=685, BW=42.7MiB/s (44.8MB/s)(5151MiB/120619msec)
slat (nsec): min=0, max=645074k, avg=32638.70, stdev=3275070.39
clat (nsec): min=0, max=2092.4M, avg=5821962.72, stdev=68390851.36
lat (nsec): min=0, max=2092.8M, avg=5855536.09, stdev=68477975.18
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
| 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
| 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
| 80.00th=[ 1112], 90.00th=[ 12480], 95.00th=[ 33024],
| 99.00th=[ 27394048], 99.50th=[ 633339904], 99.90th=[1019215872],
| 99.95th=[1199570944], 99.99th=[1518338048]
bw ( KiB/s): min= 0, max=69544, per=93.51%, avg=43953.50, stdev=36190.43, samples=2
iops : min= 0, max= 1091, avg=689.00, stdev=568.51, samples=2
lat (nsec) : 750=12.78%, 1000=65.59%
lat (usec) : 2=4.88%, 4=0.16%, 10=2.00%, 20=7.08%, 50=3.16%
lat (usec) : 100=0.23%, 250=0.11%, 500=0.14%, 750=0.15%, 1000=0.43%
lat (msec) : 2=0.94%, 4=0.68%, 10=0.41%, 20=0.18%, 50=0.21%
lat (msec) : 100=0.07%, 250=0.08%, 500=0.10%, 750=0.29%, 1000=0.23%
cpu : usr=0.16%, sys=0.65%, ctx=6935, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=82711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=51, BW=3292KiB/s (3371kB/s)(386MiB/120003msec)
slat (nsec): min=0, max=65904k, avg=36780.12, stdev=841937.14
clat (nsec): min=0, max=749872k, avg=6530086.82, stdev=44937011.76
lat (nsec): min=0, max=749907k, avg=6567813.82, stdev=44947365.91
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 956],
| 50.00th=[ 1096], 60.00th=[ 1272], 70.00th=[ 1672],
| 80.00th=[ 95744], 90.00th=[ 1368064], 95.00th=[ 14614528],
| 99.00th=[145752064], 99.50th=[425721856], 99.90th=[624951296],
| 99.95th=[666894336], 99.99th=[750780416]
bw ( KiB/s): min= 0, max= 5060, per=7.00%, avg=3292.00, stdev=2500.33, samples=2
iops : min= 0, max= 79, avg=51.00, stdev=39.60, samples=2
write: IOPS=53, BW=3395KiB/s (3477kB/s)(398MiB/120003msec)
slat (nsec): min=0, max=19783k, avg=197295.51, stdev=375928.11
clat (nsec): min=0, max=852237k, avg=68838535.89, stdev=129062909.48
lat (nsec): min=0, max=852306k, avg=69037001.18, stdev=129005986.35
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1418], 20.00th=[ 1844],
| 30.00th=[ 2442], 40.00th=[ 3458], 50.00th=[ 6718], 60.00th=[ 20841],
| 70.00th=[ 58983], 80.00th=[105382], 90.00th=[212861], 95.00th=[371196],
| 99.00th=[624952], 99.50th=[675283], 99.90th=[759170], 99.95th=[759170],
| 99.99th=[851444]
bw ( KiB/s): min= 0, max= 5222, per=73.49%, avg=3392.50, stdev=2587.30, samples=2
iops : min= 0, max= 81, avg=52.50, stdev=40.31, samples=2
lat (nsec) : 750=9.04%, 1000=12.40%
lat (usec) : 2=13.32%, 4=0.10%, 10=0.85%, 20=1.64%, 50=1.03%
lat (usec) : 100=1.11%, 250=1.43%, 500=0.82%, 750=0.69%, 1000=1.48%
lat (msec) : 2=13.19%, 4=10.76%, 10=5.76%, 20=3.66%, 50=5.25%
lat (msec) : 100=6.13%, 250=6.62%, 500=3.37%, 750=1.33%, 1000=0.06%
cpu : usr=0.08%, sys=0.29%, ctx=13113, majf=0, minf=29
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6173,6363,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=45.9MiB/s (48.1MB/s), 3292KiB/s-42.7MiB/s (3371kB/s-44.8MB/s), io=5537MiB (5806MB), run=120003-120619msec
WRITE: bw=4616KiB/s (4727kB/s), 1236KiB/s-3395KiB/s (1265kB/s-3477kB/s), io=543MiB (570MB), run=120003-120516msec
Disk stats (read/write):
rbd199: ios=4257/8892, merge=0/1, ticks=531020/899787, in_queue=1424167, util=6.02%
<example-benchmark-fio-server-771-benchmark-l7n6n>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=21, BW=1351KiB/s (1384kB/s)(159MiB/120291msec)
slat (nsec): min=0, max=412566k, avg=603490.14, stdev=11034782.35
clat (nsec): min=0, max=4347.8M, avg=188943138.26, stdev=309675547.61
lat (nsec): min=0, max=4348.0M, avg=189547874.67, stdev=309681324.16
clat percentiles (usec):
| 1.00th=[ 1139], 5.00th=[ 1778], 10.00th=[ 2311],
| 20.00th=[ 3687], 30.00th=[ 5669], 40.00th=[ 8979],
| 50.00th=[ 15270], 60.00th=[ 120062], 70.00th=[ 258999],
| 80.00th=[ 367002], 90.00th=[ 549454], 95.00th=[ 775947],
| 99.00th=[1115685], 99.50th=[1300235], 99.90th=[4328522],
| 99.95th=[4328522], 99.99th=[4328522]
bw ( KiB/s): min= 0, max= 1528, per=28.54%, avg=1351.50, stdev=249.61, samples=2
iops : min= 0, max= 23, avg=20.50, stdev= 3.54, samples=2
lat (usec) : 250=0.08%, 500=0.04%, 750=0.04%, 1000=0.16%
lat (msec) : 2=6.86%, 4=15.06%, 10=20.34%, 20=10.21%, 50=5.24%
lat (msec) : 100=1.81%, 250=9.18%, 500=19.71%, 750=6.23%, 1000=3.59%
cpu : usr=0.04%, sys=0.09%, ctx=4463, majf=0, minf=29
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2537,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=163, BW=10.2MiB/s (10.7MB/s)(1237MiB/120719msec)
slat (nsec): min=0, max=10631k, avg=33850.06, stdev=183395.68
clat (nsec): min=0, max=1826.2M, avg=24367822.86, stdev=129614967.16
lat (nsec): min=0, max=1826.2M, avg=24402732.11, stdev=129638893.04
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 756],
| 20.00th=[ 780], 30.00th=[ 804], 40.00th=[ 860],
| 50.00th=[ 892], 60.00th=[ 948], 70.00th=[ 1192],
| 80.00th=[ 1704], 90.00th=[ 3293184], 95.00th=[ 33423360],
| 99.00th=[ 784334848], 99.50th=[ 935329792], 99.90th=[1216348160],
| 99.95th=[1317011456], 99.99th=[1686110208]
bw ( KiB/s): min= 0, max=11221, per=76.60%, avg=10553.50, stdev=943.99, samples=2
iops : min= 0, max= 175, avg=164.50, stdev=14.85, samples=2
lat (nsec) : 750=9.87%, 1000=54.06%
lat (usec) : 2=16.47%, 4=0.22%, 10=0.24%, 20=0.74%, 50=0.15%
lat (usec) : 100=0.12%, 250=0.07%, 500=0.09%, 750=0.30%, 1000=1.27%
lat (msec) : 2=4.08%, 4=2.97%, 10=2.30%, 20=1.21%, 50=1.45%
lat (msec) : 100=0.61%, 250=0.52%, 500=0.91%, 750=1.16%, 1000=0.86%
cpu : usr=0.07%, sys=0.21%, ctx=7263, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=19789,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=51, BW=3305KiB/s (3385kB/s)(387MiB/120008msec)
slat (nsec): min=0, max=3588.7k, avg=25761.76, stdev=73931.98
clat (nsec): min=0, max=959944k, avg=10211143.33, stdev=51913237.46
lat (nsec): min=0, max=959982k, avg=10237801.22, stdev=51919638.05
clat percentiles (nsec):
| 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
| 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 972],
| 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1512],
| 80.00th=[ 216064], 90.00th=[ 3850240], 95.00th=[ 48496640],
| 99.00th=[325058560], 99.50th=[387973120], 99.90th=[541065216],
| 99.95th=[616562688], 99.99th=[960495616]
bw ( KiB/s): min= 0, max= 4802, per=23.99%, avg=3305.50, stdev=2116.37, samples=2
iops : min= 0, max= 75, avg=51.50, stdev=33.23, samples=2
write: IOPS=52, BW=3393KiB/s (3475kB/s)(398MiB/120008msec)
slat (nsec): min=0, max=3952.3k, avg=190717.24, stdev=251762.76
clat (nsec): min=0, max=994975k, avg=65641195.08, stdev=133569305.73
lat (nsec): min=0, max=995050k, avg=65833017.35, stdev=133524713.73
clat percentiles (usec):
| 1.00th=[ 963], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1663],
| 30.00th=[ 2073], 40.00th=[ 2835], 50.00th=[ 5211], 60.00th=[ 21365],
| 70.00th=[ 64226], 80.00th=[100140], 90.00th=[154141], 95.00th=[329253],
| 99.00th=[734004], 99.50th=[834667], 99.90th=[935330], 99.95th=[952108],
| 99.99th=[994051]
bw ( KiB/s): min= 0, max= 4852, per=71.60%, avg=3391.00, stdev=2066.17, samples=2
iops : min= 0, max= 75, avg=52.50, stdev=31.82, samples=2
lat (nsec) : 750=7.23%, 1000=13.74%
lat (usec) : 2=14.70%, 4=0.04%, 10=0.29%, 20=1.01%, 50=1.01%
lat (usec) : 100=0.84%, 250=0.68%, 500=0.45%, 750=0.44%, 1000=1.74%
lat (msec) : 2=15.66%, 4=10.29%, 10=4.86%, 20=2.95%, 50=4.81%
lat (msec) : 100=7.88%, 250=7.40%, 500=2.55%, 750=0.94%, 1000=0.50%
cpu : usr=0.07%, sys=0.30%, ctx=13776, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6198,6360,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=13.5MiB/s (14.1MB/s), 3305KiB/s-10.2MiB/s (3385kB/s-10.7MB/s), io=1624MiB (1703MB), run=120008-120719msec
WRITE: bw=4737KiB/s (4850kB/s), 1351KiB/s-3393KiB/s (1384kB/s-3475kB/s), io=556MiB (583MB), run=120008-120291msec
Disk stats (read/write):
rbd230: ios=5135/9900, merge=0/2, ticks=595516/887454, in_queue=1475455, util=4.69%
Jobs: 37 (f=37): [_(2),m(1)][99.7%][r=1206M,w=1205M][r=18.8k,w=18.8k IOPS][eta 00m:51s] <example-benchmark-fio-server-372-benchmark-lc6p4>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=25, BW=1650KiB/s (1689kB/s)(194MiB/120291msec)
slat (nsec): min=0, max=773509k, avg=852811.07, stdev=20150575.43
clat (nsec): min=0, max=2310.7M, avg=154359745.14, stdev=327539286.90
lat (nsec): min=0, max=2310.8M, avg=155213400.86, stdev=327842778.04
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1352], 10.00th=[ 1598],
| 20.00th=[ 2114], 30.00th=[ 2900], 40.00th=[ 3851],
| 50.00th=[ 5276], 60.00th=[ 8160], 70.00th=[ 17171],
| 80.00th=[ 105382], 90.00th=[ 675283], 95.00th=[ 893387],
| 99.00th=[1333789], 99.50th=[1484784], 99.90th=[2021655],
| 99.95th=[2264925], 99.99th=[2298479]
bw ( KiB/s): min= 0, max= 2336, per=36.05%, avg=1650.50, stdev=969.44, samples=2
iops : min= 0, max= 36, avg=25.50, stdev=14.85, samples=2
lat (usec) : 500=0.10%, 750=0.13%, 1000=0.55%
lat (msec) : 2=17.43%, 4=22.82%, 10=23.08%, 20=7.00%, 50=3.52%
lat (msec) : 100=5.07%, 250=1.94%, 500=2.61%, 750=8.20%, 1000=4.07%
cpu : usr=0.03%, sys=0.12%, ctx=5780, majf=0, minf=27
IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3098,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
read: IOPS=610, BW=38.0MiB/s (39.9MB/s)(4588MiB/120573msec)
slat (nsec): min=0, max=771813k, avg=42177.78, stdev=4068032.92
clat (nsec): min=0, max=1795.7M, avg=6503281.36, stdev=66799159.40
lat (nsec): min=0, max=1795.8M, avg=6546418.17, stdev=66930745.83
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
| 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 884],
| 80.00th=[ 972], 90.00th=[ 11840], 95.00th=[ 995328],
| 99.00th=[ 91750400], 99.50th=[ 599785472], 99.90th=[ 952107008],
| 99.95th=[1115684864], 99.99th=[1400897536]
bw ( KiB/s): min= 0, max=63106, per=93.37%, avg=39021.00, stdev=34061.33, samples=2
iops : min= 0, max= 990, avg=611.50, stdev=535.28, samples=2
lat (nsec) : 750=15.86%, 1000=64.91%
lat (usec) : 2=5.86%, 4=0.14%, 10=1.15%, 20=5.13%, 50=0.69%
lat (usec) : 100=0.19%, 250=0.10%, 500=0.19%, 750=0.28%, 1000=0.49%
lat (msec) : 2=1.20%, 4=1.00%, 10=0.73%, 20=0.33%, 50=0.46%
lat (msec) : 100=0.31%, 250=0.10%, 500=0.23%, 750=0.38%, 1000=0.18%
cpu : usr=0.15%, sys=0.61%, ctx=8969, majf=0, minf=31
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=73655,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
read: IOPS=44, BW=2840KiB/s (2908kB/s)(333MiB/120025msec)
slat (nsec): min=0, max=2616.8k, avg=30791.64, stdev=88068.41
clat (nsec): min=0, max=642034k, avg=7264005.10, stdev=39713153.45
lat (nsec): min=0, max=642063k, avg=7295930.78, stdev=39719952.99
clat percentiles (nsec):
| 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
| 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 988],
| 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 9920],
| 80.00th=[ 684032], 90.00th=[ 2899968], 95.00th=[ 41680896],
| 99.00th=[181403648], 99.50th=[362807296], 99.90th=[501219328],
| 99.95th=[530579456], 99.99th=[641728512]
bw ( KiB/s): min= 0, max= 4536, per=6.80%, avg=2840.00, stdev=2398.51, samples=2
iops : min= 0, max= 70, avg=43.50, stdev=37.48, samples=2
write: IOPS=45, BW=2935KiB/s (3005kB/s)(344MiB/120025msec)
slat (nsec): min=0, max=5247.4k, avg=183124.89, stdev=260290.31
clat (nsec): min=0, max=922637k, avg=80167211.55, stdev=145551293.96
lat (nsec): min=0, max=922680k, avg=80351609.68, stdev=145505979.46
clat percentiles (usec):
| 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1434], 20.00th=[ 1860],
| 30.00th=[ 2573], 40.00th=[ 3916], 50.00th=[ 8979], 60.00th=[ 28705],
| 70.00th=[ 68682], 80.00th=[113771], 90.00th=[278922], 95.00th=[450888],
| 99.00th=[650118], 99.50th=[725615], 99.90th=[859833], 99.95th=[918553],
| 99.99th=[918553]
bw ( KiB/s): min= 0, max= 4651, per=64.06%, avg=2932.50, stdev=2430.33, samples=2
iops : min= 0, max= 72, avg=45.00, stdev=38.18, samples=2
lat (nsec) : 750=7.63%, 1000=12.33%
lat (usec) : 2=13.99%, 4=0.14%, 10=0.38%, 20=0.85%, 50=0.85%
lat (usec) : 100=0.89%, 250=1.16%, 500=0.70%, 750=0.79%, 1000=1.73%
lat (msec) : 2=13.41%, 4=10.59%, 10=5.92%, 20=3.28%, 50=5.86%
lat (msec) : 100=7.03%, 250=6.65%, 500=3.85%, 750=1.83%, 1000=0.17%
cpu : usr=0.06%, sys=0.26%, ctx=12141, majf=0, minf=30
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5326,5501,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=40.8MiB/s (42.8MB/s), 2840KiB/s-38.0MiB/s (2908kB/s-39.9MB/s), io=4921MiB (5160MB), run=120025-120573msec
WRITE: bw=4578KiB/s (4688kB/s), 1650KiB/s-2935KiB/s (1689kB/s-3005kB/s), io=538MiB (564MB), run=120025-120291msec
Disk stats (read/write):
rbd130: ios=6082/9151, merge=0/1, ticks=546406/923365, in_queue=1462130, util=7.65%
<example-benchmark-fio-server-452-benchmark-7lt6f>
randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
write: IOPS=45, BW=2916KiB/s (2986kB/s)(346MiB/121551msec)
slat (nsec): min=0, max=8601.9M, avg=1768759.47, stdev=115624208.02
clat (nsec): min=0, max=11024M, avg=85317961.64, stdev=306099695.43
lat (nsec): min=0, max=11024M, avg=87087195.73, stdev=327190446.01
clat percentiles (usec):
| 1.00th=[ 914], 5.00th=[ 1745], 10.00th=[ 2507],
| 20.00th=[ 3720], 30.00th=[ 5145], 40.00th=[ 7504],
| 50.00th=[ 10421], 60.00th=[ 15533], 70.00th=[ 29492],
| 80.00th=[ 102237], 90.00th=[ 219153], 95.00th=[ 484443],
| 99.00th=[ 759170], 99.50th=[ 935330], 99.90th=[ 3305112],
| 99.95th=[ 8657044], 99.99th=[11072963]
bw ( KiB/s): min= 0, max= 3061, per=47.12%, avg=2950.00, stdev=156.98, samples=2
iops : min= 0, max= 47, avg=45.50, stdev= 2.12, samples=2
lat (usec) : 4=0.04%, 50=0.02%, 250=0.07%, 500=0.04%, 750=0.33%
lat (usec) : 1000=0.89%
lat (msec) : 2=5.53%, 4=15.37%, 10=26.41%, 20=16.22%, 50=9.83%