liu-chunmei/seastore nbd test.md

## seastore nbd test.md

      
    Raw
  

              seastore nbd test.md
            
          
    fio:  seastore nbd

Scenario


seastore transaction manager running as nbd server
fio nbd engine.

fio test result: (release version)

raw disk /dev/nvme1n1
         iodepth=1
numjobs=1 r: 12.31(us)  w: 12.78(us)
numjobs=2 r: 14.39(us)  w: 14.78(us)
note: numjobs=2, and set two jobs.

         iodepth=1                                     iodepth=16                         iodepth=32 
numjobs=1  r: 49.84(us)  w: 107.26(us)              r: 277.82(us)  w: 1230.33(us)       r: 604.24(us)  w: 3473.96(us)
numjobs=2  r: 60.65(us)  w: 97.14(us)               r: 385.84      w: 879.91(us)        r: 813.60(us)  w: 1706.96(us)
note: numjobs=2, and set two jobs.

install fio nbd

download libnbd: libnbd-1.6 stable

http://download.libguestfs.org/libnbd/
./configure
make & make install
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"

download fio: fio-3.25-97-gc4f5

git clone http://git.kernel.dk/fio.git
./configure --enable-libnbd
make & make install

version

master (commit ID a3591378a59c621ac597987facb3fb30707c218f)

crimson Build configuration

./do_cmake.sh -DWITH_SEASTAR=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS="-fno-omit-frame-pointer" -DWITH_TESTS=OFF

running seastore nbd server

sudo ./bin/crimson-store-nbd --device-path /dev/nvme1n1 -c 1 --total-device-size=107374182400 --mkfs true --uds-path /tmp/store_nbd_socket.sock --memory 64G 

with debug (--debug on)enables debug output
strace -tt -f ./bin/crimson-store-nbd to trace system error
strace -f -tt -p <pid>

fio configuration  (nbd.fio)

 [global]
   ioengine=nbd
   uri=nbd+unix:///?socket=/tmp/store_nbd_socket.sock
   rw=randrw
   time_based
   runtime=900
   group_reporting
   iodepth=1
   size=500G
   numjobs=1
 
   [job0]
   offset=0
   
   [job1]
   offset=250M  //when numjobs=2

run fio test

$ sudo fio nbd.fio
fio result: using release build

iodepth=1 numjobs=1

job0: (groupid=0, jobs=1): err= 0: pid=61932: Thu Mar 25 15:56:39 2021
  read: IOPS=6280, BW=24.5MiB/s (25.7MB/s)(21.6GiB/900118msec)
    slat (nsec): min=2306, max=419176, avg=4311.84, stdev=744.21
    clat (usec): min=10, max=24379, avg=45.53, stdev=139.72
     lat (usec): min=21, max=24383, avg=49.84, stdev=139.72
    clat percentiles (usec):
     |  1.00th=[   22],  5.00th=[   24], 10.00th=[   26], 20.00th=[   39],
     | 30.00th=[   40], 40.00th=[   41], 50.00th=[   44], 60.00th=[   46],
     | 70.00th=[   48], 80.00th=[   52], 90.00th=[   57], 95.00th=[   60],
     | 99.00th=[   68], 99.50th=[   75], 99.90th=[  562], 99.95th=[  938],
     | 99.99th=[ 5080]
   bw (  KiB/s): min=11927, max=49416, per=100.00%, avg=25138.58, stdev=6094.67, samples=1799
   iops        : min= 2981, max=12354, avg=6284.59, stdev=1523.69, samples=1799
  write: IOPS=6275, BW=24.5MiB/s (25.7MB/s)(21.5GiB/900118msec); 0 zone resets
    slat (usec): min=4, max=410, avg= 8.18, stdev= 1.00
    clat (usec): min=7, max=278164, avg=99.08, stdev=2092.88
     lat (usec): min=46, max=278172, avg=107.26, stdev=2092.88
    clat percentiles (usec):
     |  1.00th=[    45],  5.00th=[    48], 10.00th=[    53], 20.00th=[    62],
     | 30.00th=[    63], 40.00th=[    67], 50.00th=[    70], 60.00th=[    73],
     | 70.00th=[    77], 80.00th=[    82], 90.00th=[    88], 95.00th=[    93],
     | 99.00th=[   108], 99.50th=[   126], 99.90th=[   979], 99.95th=[  1483],
     | 99.99th=[141558]
   bw (  KiB/s): min=11984, max=49576, per=100.00%, avg=25118.11, stdev=6079.67, samples=1799
   iops        : min= 2996, max=12394, avg=6279.47, stdev=1519.93, samples=1799
  lat (usec)   : 10=0.01%, 20=0.02%, 50=42.39%, 100=56.47%, 250=0.86%
  lat (usec)   : 500=0.10%, 750=0.05%, 1000=0.04%
  lat (msec)   : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=12.75%, sys=15.74%, ctx=11302995, majf=0, minf=1019
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5653465,5648844,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=24.5MiB/s (25.7MB/s), 24.5MiB/s-24.5MiB/s (25.7MB/s-25.7MB/s), io=21.6GiB (23.2GB), run=900118-900118msec
  WRITE: bw=24.5MiB/s (25.7MB/s), 24.5MiB/s-24.5MiB/s (25.7MB/s-25.7MB/s), io=21.5GiB (23.1GB), run=900118-900118msec

...

iodepth=1 numjobs=2

job0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iode         pth=1
...
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iode         pth=1
...
fio-3.25-97-gc4f5
Starting 4 processes
fio: connected to NBD server
fio: connected to NBD server[r=20.9MiB/s,w=21.1MiB/s][r=5340,w=5394 IOPS][eta 15m:00s]
fio: connected to NBD server6.7%][r=23.4MiB/s,w=27.7MiB/s][r=5981,w=7079 IOPS][eta 15m:00s]
fio: connected to NBD server1),m(1)][60.0%][r=24.0MiB/s,w=28.6MiB/s][r=6133,w=7332 IOPS][eta 30m:00s]
Jobs: 1 (f=1): [_(1),m(1),_(2)][57.1%][r=10.8MiB/s,w=11.0MiB/s][r=2774,w=2817 IOPS][eta 45m:00s]
job0: (groupid=0, jobs=4): err= 0: pid=62521: Thu Mar 25 17:45:06 2021
  read: IOPS=23.3k, BW=91.1MiB/s (95.5MB/s)(80.0GiB/900001msec)
    slat (nsec): min=1614, max=504034, avg=3925.76, stdev=760.15
    clat (usec): min=10, max=38062, avg=56.72, stdev=98.01
     lat (usec): min=18, max=38067, avg=60.65, stdev=98.02
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   31], 10.00th=[   40], 20.00th=[   43],
     | 30.00th=[   46], 40.00th=[   50], 50.00th=[   55], 60.00th=[   59],
     | 70.00th=[   62], 80.00th=[   66], 90.00th=[   73], 95.00th=[   79],
     | 99.00th=[   94], 99.50th=[  101], 99.90th=[  938], 99.95th=[ 1811],
     | 99.99th=[ 2409]
   bw (  KiB/s): min=33360, max=136280, per=100.00%, avg=93300.76, stdev=4001.23, samples=7195
   iops        : min= 8340, max=34070, avg=23325.03, stdev=1000.31, samples=7195
  write: IOPS=25.7k, BW=100MiB/s (105MB/s)(88.1GiB/900001msec); 0 zone resets
    slat (usec): min=3, max=415, avg= 7.83, stdev= 1.11
    clat (usec): min=8, max=517456, avg=89.31, stdev=1868.34
     lat (usec): min=42, max=517464, avg=97.14, stdev=1868.34
    clat percentiles (usec):
     |  1.00th=[    45],  5.00th=[    47], 10.00th=[    49], 20.00th=[    52],
     | 30.00th=[    55], 40.00th=[    59], 50.00th=[    63], 60.00th=[    67],
     | 70.00th=[    73], 80.00th=[    80], 90.00th=[    89], 95.00th=[    95],
     | 99.00th=[   113], 99.50th=[   122], 99.90th=[  1287], 99.95th=[  1926],
     | 99.99th=[104334]
   bw (  KiB/s): min=37136, max=147432, per=100.00%, avg=102729.95, stdev=4136.90, samples=7195
   iops        : min= 9284, max=36858, avg=25682.32, stdev=1034.23, samples=7195
  lat (usec)   : 10=0.01%, 20=0.01%, 50=27.87%, 100=70.37%, 250=1.54%
  lat (usec)   : 500=0.05%, 750=0.03%, 1000=0.02%
  lat (msec)   : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=12.65%, sys=15.11%, ctx=44086958, majf=0, minf=5583
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=20981499,23102486,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=91.1MiB/s (95.5MB/s), 91.1MiB/s-91.1MiB/s (95.5MB/s-95.5MB/s), io=80.0GiB (85.9GB), run=900001-900001msec
  WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=88.1GiB (94.6GB), run=900001-900001msec


iopdeth=16 numjobs=1

job0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=16
fio-3.25-97-gc4f5
Starting 1 process
fio: connected to NBD server
Jobs: 1 (f=1): [m(1)][100.0%][r=23.9MiB/s,w=23.2MiB/s][r=6117,w=5939 IOPS][eta 00m:00s]
job0: (groupid=0, jobs=1): err= 0: pid=62018: Thu Mar 25 16:17:38 2021
  read: IOPS=10.6k, BW=41.4MiB/s (43.4MB/s)(36.4GiB/900044msec)
    slat (nsec): min=1551, max=318093, avg=2858.23, stdev=680.73
    clat (usec): min=14, max=33182, avg=274.96, stdev=441.81
     lat (usec): min=17, max=33186, avg=277.82, stdev=441.89
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[  133], 10.00th=[  186], 20.00th=[  208],
     | 30.00th=[  223], 40.00th=[  235], 50.00th=[  247], 60.00th=[  262],
     | 70.00th=[  277], 80.00th=[  306], 90.00th=[  359], 95.00th=[  396],
     | 99.00th=[  486], 99.50th=[  676], 99.90th=[ 7701], 99.95th=[11207],
     | 99.99th=[16581]
   bw (  KiB/s): min= 5880, max=74864, per=100.00%, avg=42440.21, stdev=17003.05, samples=1799
   iops        : min= 1470, max=18716, avg=10610.00, stdev=4250.78, samples=1799
  write: IOPS=10.6k, BW=41.4MiB/s (43.4MB/s)(36.4GiB/900044msec); 0 zone resets
    slat (usec): min=3, max=432, avg= 5.93, stdev= 1.17
    clat (usec): min=100, max=668424, avg=1224.40, stdev=13303.30
     lat (usec): min=104, max=668430, avg=1230.33, stdev=13303.31
    clat percentiles (usec):
     |  1.00th=[   200],  5.00th=[   265], 10.00th=[   302], 20.00th=[   326],
     | 30.00th=[   343], 40.00th=[   355], 50.00th=[   363], 60.00th=[   375],
     | 70.00th=[   388], 80.00th=[   400], 90.00th=[   424], 95.00th=[   445],
     | 99.00th=[   955], 99.50th=[ 62129], 99.90th=[250610], 99.95th=[291505],
     | 99.99th=[429917]
   bw (  KiB/s): min= 5968, max=73928, per=100.00%, avg=42422.36, stdev=16973.17, samples=1799
   iops        : min= 1492, max=18482, avg=10605.54, stdev=4243.32, samples=1799
  lat (usec)   : 20=0.01%, 50=0.09%, 100=1.42%, 250=26.51%, 500=70.30%
  lat (usec)   : 750=0.91%, 1000=0.13%
  lat (msec)   : 2=0.10%, 4=0.04%, 10=0.12%, 20=0.06%, 50=0.01%
  lat (msec)   : 100=0.22%, 250=0.06%, 500=0.05%, 750=0.01%
  cpu          : usr=12.20%, sys=13.68%, ctx=3985339, majf=0, minf=987
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=9541361,9537360,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=36.4GiB (39.1GB), run=900044-900044msec
  WRITE: bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=36.4GiB (39.1GB), run=900044-900044msec


iodepth=16 numjobs=2

job0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=16
...
job01: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=16
...
fio-3.25-97-gc4f5
Starting 4 processes
fio: connected to NBD server
fio: connected to NBD server[r=21.1MiB/s,w=21.7MiB/s][r=5409,w=5561 IOPS][eta 15m:00s]
fio: connected to NBD server6.7%][r=64.7MiB/s,w=77.4MiB/s][r=16.6k,w=19.8k IOPS][eta 15m:00s]
fio: connected to NBD server1)][60.0%][r=67.6MiB/s,w=80.0MiB/s][r=17.3k,w=20.5k IOPS][eta 30m:00s]
Jobs: 1 (f=1): [_(1),m(1),_(2)][57.1%][r=23.6MiB/s,w=24.1MiB/s][r=6045,w=6171 IOPS][eta 45m:00s]
job0: (groupid=0, jobs=4): err= 0: pid=62845: Thu Mar 25 19:08:48 2021
  read: IOPS=44.5k, BW=174MiB/s (182MB/s)(153GiB/900171msec)
    slat (nsec): min=950, max=475994, avg=2757.32, stdev=773.98
    clat (usec): min=16, max=431503, avg=383.08, stdev=1327.37
     lat (usec): min=19, max=431506, avg=385.84, stdev=1327.38
    clat percentiles (usec):
     |  1.00th=[  151],  5.00th=[  208], 10.00th=[  231], 20.00th=[  269],
     | 30.00th=[  297], 40.00th=[  322], 50.00th=[  347], 60.00th=[  367],
     | 70.00th=[  392], 80.00th=[  420], 90.00th=[  461], 95.00th=[  490],
     | 99.00th=[  578], 99.50th=[  758], 99.90th=[ 7767], 99.95th=[24249],
     | 99.99th=[68682]
   bw (  KiB/s): min=61515, max=288776, per=100.00%, avg=180234.95, stdev=8754.40, samples=7024
   iops        : min=15378, max=72194, avg=45058.44, stdev=2188.60, samples=7024
  write: IOPS=50.7k, BW=198MiB/s (208MB/s)(174GiB/900171msec); 0 zone resets
    slat (usec): min=2, max=426, avg= 5.72, stdev= 1.26
    clat (usec): min=61, max=961852, avg=874.19, stdev=13946.44
     lat (usec): min=67, max=961858, avg=879.91, stdev=13946.46
    clat percentiles (usec):
     |  1.00th=[   151],  5.00th=[   178], 10.00th=[   192], 20.00th=[   210],
     | 30.00th=[   223], 40.00th=[   233], 50.00th=[   245], 60.00th=[   260],
     | 70.00th=[   281], 80.00th=[   334], 90.00th=[   383], 95.00th=[   412],
     | 99.00th=[   627], 99.50th=[  8717], 99.90th=[168821], 99.95th=[362808],
     | 99.99th=[624952]
   bw (  KiB/s): min=73034, max=320504, per=100.00%, avg=204871.54, stdev=9284.60, samples=7024
   iops        : min=18258, max=80126, avg=51217.56, stdev=2321.16, samples=7024
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.21%, 250=35.74%, 500=61.45%
  lat (usec)   : 750=1.90%, 1000=0.11%
  lat (msec)   : 2=0.13%, 4=0.09%, 10=0.07%, 20=0.04%, 50=0.10%
  lat (msec)   : 100=0.08%, 250=0.03%, 500=0.03%, 750=0.01%, 1000=0.01%
  cpu          : usr=15.16%, sys=15.19%, ctx=19000313, majf=0, minf=7740
  IO depths    : 1=0.8%, 2=1.6%, 4=3.3%, 8=6.6%, 16=87.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.6%, 8=0.0%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40063553,45604369,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=153GiB (164GB), run=900171-900171msec
  WRITE: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=174GiB (187GB), run=900171-900171msec


iodepth=32 numjobs=1

job0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=32
fio-3.25-97-gc4f5
Starting 1 process
fio: connected to NBD server
Jobs: 1 (f=1): [m(1)][100.0%][r=17.2MiB/s,w=17.4MiB/s][r=4397,w=4445 IOPS][eta 00m:00s]
job0: (groupid=0, jobs=1): err= 0: pid=62646: Thu Mar 25 18:06:11 2021
  read: IOPS=7846, BW=30.7MiB/s (32.1MB/s)(26.9GiB/900001msec)
    slat (nsec): min=1313, max=236301, avg=2944.75, stdev=705.63
    clat (usec): min=17, max=75288, avg=601.30, stdev=808.05
     lat (usec): min=19, max=75291, avg=604.24, stdev=808.14
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  326], 10.00th=[  388], 20.00th=[  437],
     | 30.00th=[  469], 40.00th=[  502], 50.00th=[  537], 60.00th=[  578],
     | 70.00th=[  635], 80.00th=[  685], 90.00th=[  758], 95.00th=[  807],
     | 99.00th=[ 1123], 99.50th=[ 3163], 99.90th=[12911], 99.95th=[16909],
     | 99.99th=[29492]
   bw (  KiB/s): min=  272, max=71184, per=100.00%, avg=31605.36, stdev=14278.72, samples=1788
   iops        : min=   68, max=17796, avg=7901.28, stdev=3569.70, samples=1788
  write: IOPS=7842, BW=30.6MiB/s (32.1MB/s)(26.9GiB/900001msec); 0 zone resets
    slat (usec): min=2, max=334, avg= 6.19, stdev= 1.16
    clat (usec): min=177, max=777948, avg=3467.77, stdev=27250.00
     lat (usec): min=183, max=777954, avg=3473.96, stdev=27250.04
    clat percentiles (usec):
     |  1.00th=[   383],  5.00th=[   510], 10.00th=[   570], 20.00th=[   611],
     | 30.00th=[   635], 40.00th=[   660], 50.00th=[   676], 60.00th=[   701],
     | 70.00th=[   725], 80.00th=[   758], 90.00th=[   807], 95.00th=[   898],
     | 99.00th=[ 96994], 99.50th=[223347], 99.90th=[421528], 99.95th=[463471],
     | 99.99th=[574620]
   bw (  KiB/s): min=  272, max=71232, per=100.00%, avg=31589.84, stdev=14249.06, samples=1788
   iops        : min=   68, max=17810, avg=7897.40, stdev=3562.29, samples=1788
  lat (usec)   : 20=0.01%, 50=0.02%, 100=0.14%, 250=1.54%, 500=20.35%
  lat (usec)   : 750=61.83%, 1000=13.63%
  lat (msec)   : 2=1.03%, 4=0.20%, 10=0.28%, 20=0.15%, 50=0.09%
  lat (msec)   : 100=0.33%, 250=0.22%, 500=0.19%, 750=0.02%, 1000=0.01%
  cpu          : usr=9.35%, sys=10.06%, ctx=1639939, majf=0, minf=86
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=7062091,7058642,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=30.7MiB/s (32.1MB/s), 30.7MiB/s-30.7MiB/s (32.1MB/s-32.1MB/s), io=26.9GiB (28.9GB), run=900001-900001msec
  WRITE: bw=30.6MiB/s (32.1MB/s), 30.6MiB/s-30.6MiB/s (32.1MB/s-32.1MB/s), io=26.9GiB (28.9GB), run=900001-900001msec


iodepth=32 numjobs=2

job0: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=32
...
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=nbd, iodepth=32
...
fio-3.25-97-gc4f5
Starting 4 processes
fio: connected to NBD server
fio: connected to NBD server[r=22.7MiB/s,w=22.7MiB/s][r=5823,w=5818 IOPS][eta 15m:00s]
fio: connected to NBD server6.7%][r=67.5MiB/s,w=79.9MiB/s][r=17.3k,w=20.5k IOPS][eta 15m:00s]
fio: connected to NBD server1),m(1)][60.0%][r=69.1MiB/s,w=82.6MiB/s][r=17.7k,w=21.1k IOPS][eta 30m:00s]
Jobs: 1 (f=1): [_(1),m(1),_(2)][57.1%][r=12.0MiB/s,w=12.3MiB/s][r=3067,w=3139 IOPS][eta 45m:00s]
job0: (groupid=0, jobs=4): err= 0: pid=5563: Fri Mar 26 15:44:46 2021
  read: IOPS=44.2k, BW=173MiB/s (181MB/s)(152GiB/900085msec)
    slat (nsec): min=1045, max=427980, avg=2814.80, stdev=790.83
    clat (usec): min=17, max=473149, avg=810.79, stdev=2619.12
     lat (usec): min=20, max=473153, avg=813.60, stdev=2619.11
    clat percentiles (usec):
     |  1.00th=[  277],  5.00th=[  371], 10.00th=[  416], 20.00th=[  478],
     | 30.00th=[  537], 40.00th=[  586], 50.00th=[  635], 60.00th=[  676],
     | 70.00th=[  725], 80.00th=[  783], 90.00th=[  873], 95.00th=[  971],
     | 99.00th=[ 1270], 99.50th=[16319], 99.90th=[36439], 99.95th=[56361],
     | 99.99th=[83362]
   bw (  KiB/s): min=45128, max=286233, per=100.00%, avg=179069.87, stdev=8712.08, samples=7036
   iops        : min=11282, max=71558, avg=44767.27, stdev=2178.02, samples=7036
  write: IOPS=50.5k, BW=197MiB/s (207MB/s)(173GiB/900085msec); 0 zone resets
    slat (usec): min=2, max=438, avg= 5.92, stdev= 1.33
    clat (usec): min=122, max=930652, avg=1701.03, stdev=19792.70
     lat (usec): min=127, max=930658, avg=1706.96, stdev=19792.72
    clat percentiles (usec):
     |  1.00th=[   258],  5.00th=[   322], 10.00th=[   359], 20.00th=[   392],
     | 30.00th=[   416], 40.00th=[   437], 50.00th=[   461], 60.00th=[   494],
     | 70.00th=[   562], 80.00th=[   652], 90.00th=[   717], 95.00th=[   783],
     | 99.00th=[ 18482], 99.50th=[ 42206], 99.90th=[367002], 99.95th=[509608],
     | 99.99th=[683672]
   bw (  KiB/s): min=53832, max=315579, per=100.00%, avg=204235.53, stdev=9133.75, samples=7034
   iops        : min=13458, max=78896, avg=51058.67, stdev=2283.44, samples=7034
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.03%, 250=0.80%, 500=43.30%
  lat (usec)   : 750=40.05%, 1000=12.88%
  lat (msec)   : 2=1.88%, 4=0.11%, 10=0.06%, 20=0.19%, 50=0.43%
  lat (msec)   : 100=0.11%, 250=0.07%, 500=0.06%, 750=0.03%, 1000=0.01%
  cpu          : usr=15.74%, sys=14.38%, ctx=10811771, majf=0, minf=12543
  IO depths    : 1=0.8%, 2=1.7%, 4=3.4%, 8=6.7%, 16=13.4%, 32=74.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.5%, 8=0.0%, 16=0.0%, 32=0.5%, 64=0.0%, >=64=0.0%
     issued rwts: total=39815802,45470963,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=173MiB/s (181MB/s), 173MiB/s-173MiB/s (181MB/s-181MB/s), io=152GiB (163GB), run=900085-900085msec
  WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=173GiB (186GB), run=900085-900085msec


## second run still alive.txt
ERROR 2021-02-17 22:23:39,827 [shard 0] filestore - ~Cache: extent CachedExtent(addr=0x6010000c8d20, type=ROOT, version=0, dirty_from=journal_seq_t(segment_seq=0, offset=paddr_t<0, 4096>), paddr=paddr_t<NULL_SEG, NULL_OFF>, state=DIRTY, last_committed_crc=0, refcount=2) still alive
ERROR 2021-02-17 22:23:39,828 [shard 0] none - /home/chunmei/ceph/src/crimson/os/seastore/cache.cc:32 : In function 'crimson::os::seastore::Cache::~Cache()', ceph_assert(%s)
extents.empty()
Aborting on shard 0.
Backtrace:
  0x0000000000848de9
  0x000000000080d575
  0x000000000080d85a
  0x000000000080d932
  /lib/x86_64-linux-gnu/libpthread.so.0+0x000000000001289f
  /lib/x86_64-linux-gnu/libc.so.6+0x000000000003ef46
  /lib/x86_64-linux-gnu/libc.so.6+0x00000000000408b0
  0x00000000002f620c
  0x00000000002f624d

  need remove /tmp/store_nbd_socket.sock  then run again.
  0x00000000003e3150
  0x000000000035a2a0
  0x000000000080a050
  0x000000000080a3d6
  0x0000000000838f75
  0x00000000007d25be
  0x00000000007d3158
  0x00000000007d3288
  0x000000000035801e
  /usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x00000000000d086f
  /lib/x86_64-linux-gnu/libpthread.so.0+0x00000000000076da
  /lib/x86_64-linux-gnu/libc.so.6+0x0000000000121a3e

sudo rm /tmp/store_nbd_socket.sock to solve it.
	ERROR 2021-02-17 22:23:39,827 [shard 0] filestore - ~Cache: extent CachedExtent(addr=0x6010000c8d20, type=ROOT, version=0, dirty_from=journal_seq_t(segment_seq=0, offset=paddr_t<0, 4096>), paddr=paddr_t<NULL_SEG, NULL_OFF>, state=DIRTY, last_committed_crc=0, refcount=2) still alive
	ERROR 2021-02-17 22:23:39,828 [shard 0] none - /home/chunmei/ceph/src/crimson/os/seastore/cache.cc:32 : In function 'crimson::os::seastore::Cache::~Cache()', ceph_assert(%s)
	extents.empty()
	Aborting on shard 0.
	Backtrace:
	0x0000000000848de9
	0x000000000080d575
	0x000000000080d85a
	0x000000000080d932
	/lib/x86_64-linux-gnu/libpthread.so.0+0x000000000001289f
	/lib/x86_64-linux-gnu/libc.so.6+0x000000000003ef46
	/lib/x86_64-linux-gnu/libc.so.6+0x00000000000408b0
	0x00000000002f620c
	0x00000000002f624d

	need remove /tmp/store_nbd_socket.sock then run again.
	0x00000000003e3150
	0x000000000035a2a0
	0x000000000080a050
	0x000000000080a3d6
	0x0000000000838f75
	0x00000000007d25be
	0x00000000007d3158
	0x00000000007d3288
	0x000000000035801e
	/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x00000000000d086f
	/lib/x86_64-linux-gnu/libpthread.so.0+0x00000000000076da
	/lib/x86_64-linux-gnu/libc.so.6+0x0000000000121a3e

	sudo rm /tmp/store_nbd_socket.sock to solve it.