Performance tests of commit 92a28f033a5272b7dc2c5d726e67b6d09f6166ba

with.txt

$ /work/fio/fio /work/fio/examples/rbd.fio 
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=17.4MiB/s][r=0,w=4467 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=28369: Wed Oct 11 15:16:59 2017
  write: IOPS=4321, BW=16.9MiB/s (17.7MB/s)(4096MiB/242614msec)
    slat (nsec): min=626, max=1643.6k, avg=2736.83, stdev=6738.13
    clat (usec): min=692, max=254925, avg=7399.87, stdev=3855.18
     lat (usec): min=694, max=254927, avg=7402.61, stdev=3854.96
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   11], 95.00th=[   12],
     | 99.00th=[   15], 99.50th=[   20], 99.90th=[   35], 99.95th=[   45],
     | 99.99th=[  155]
   bw (  KiB/s): min= 5984, max=19168, per=99.99%, avg=17284.91, stdev=1377.36, samples=485
   iops        : min= 1496, max= 4792, avg=4321.21, stdev=344.34, samples=485
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.46%, 4=6.79%, 10=82.03%, 20=10.25%, 50=0.43%
  lat (msec)   : 100=0.02%, 250=0.03%, 500=0.01%
  cpu          : usr=29.67%, sys=67.25%, ctx=553921, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=16.9MiB/s (17.7MB/s), 16.9MiB/s-16.9MiB/s (17.7MB/s-17.7MB/s), io=4096MiB (4295MB), run=242614-242614msec

Disk stats (read/write):
  nvme0n1: ios=1252/4975954, merge=13/424995, ticks=2084/430156, in_queue=433160, util=31.44%

without.txt

$ /work/fio/fio /work/fio/examples/rbd.fio 
rbd_iodepth32: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.0-32-g83070
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=17.3MiB/s][r=0,w=4439 IOPS][eta 00m:00s]
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=31242: Wed Oct 11 15:23:28 2017
  write: IOPS=4330, BW=16.9MiB/s (17.7MB/s)(4096MiB/242124msec)
    slat (nsec): min=629, max=3154.4k, avg=2888.35, stdev=8514.67
    clat (usec): min=774, max=261686, avg=7384.65, stdev=3653.51
     lat (usec): min=780, max=261688, avg=7387.54, stdev=3653.36
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[   10], 95.00th=[   11],
     | 99.00th=[   15], 99.50th=[   20], 99.90th=[   36], 99.95th=[   46],
     | 99.99th=[  159]
   bw (  KiB/s): min= 5560, max=19224, per=99.99%, avg=17319.82, stdev=1396.23, samples=484
   iops        : min= 1390, max= 4806, avg=4329.94, stdev=349.06, samples=484
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.10%, 4=3.66%, 10=87.39%, 20=8.39%, 50=0.42%
  lat (msec)   : 100=0.02%, 250=0.02%, 500=0.01%
  cpu          : usr=28.93%, sys=66.63%, ctx=586278, majf=0, minf=5
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=16.9MiB/s (17.7MB/s), 16.9MiB/s-16.9MiB/s (17.7MB/s-17.7MB/s), io=4096MiB (4295MB), run=242124-242124msec

Disk stats (read/write):
  nvme0n1: ios=1244/4980509, merge=12/424736, ticks=1684/426412, in_queue=428648, util=31.78%