Skip to content

Instantly share code, notes, and snippets.

@mittachaitu

mittachaitu/LVM_Snapshot_performance

Created October 14, 2022 19:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mittachaitu/17006a1e3f08ad0c17add7ea007e63b6 to your computer and use it in GitHub Desktop.
Save mittachaitu/17006a1e3f08ad0c17add7ea007e63b6 to your computer and use it in GitHub Desktop.
Download ZIP
Checking Performance of Thin LVM volume by creating and deleting thin snapshot without writing data to volume after snapshot creation
Raw
ext4_perf.txt
Running PRE-SNAPSHOT_RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152 8325/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 1065600 209715199
PRE-SNAPSHOT_RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-1: (groupid=0, jobs=3): err= 0: pid=5603: Fri Oct 14 18:07:52 2022
read: IOPS=2262, BW=35.4MiB/s (37.1MB/s)(4245MiB/120088msec)
slat (usec): min=3, max=151584, avg=69.17, stdev=2346.53
clat (nsec): min=800, max=154462k, avg=2072371.69, stdev=8203371.12
lat (usec): min=8, max=154471, avg=2141.72, stdev=8528.67
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 23],
| 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 32], 60.00th=[ 42],
| 70.00th=[ 1680], 80.00th=[ 2008], 90.00th=[ 2638], 95.00th=[ 4228],
| 99.00th=[ 37487], 99.50th=[ 43779], 99.90th=[110625], 99.95th=[143655],
| 99.99th=[152044]
bw ( KiB/s): min=26400, max=69504, per=100.00%, avg=36226.67, stdev=1633.10, samples=720
iops : min= 1650, max= 4344, avg=2264.17, stdev=102.07, samples=720
write: IOPS=2257, BW=35.3MiB/s (37.0MB/s)(4237MiB/120088msec); 0 zone resets
slat (usec): min=3, max=151500, avg=64.76, stdev=2282.15
clat (usec): min=720, max=157875, avg=8414.82, stdev=14823.72
lat (usec): min=749, max=157884, avg=8479.75, stdev=14992.22
clat percentiles (usec):
| 1.00th=[ 873], 5.00th=[ 963], 10.00th=[ 1037], 20.00th=[ 1237],
| 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2540], 60.00th=[ 2737],
| 70.00th=[ 3064], 80.00th=[ 5080], 90.00th=[ 34341], 95.00th=[ 38011],
| 99.00th=[ 50594], 99.50th=[ 64226], 99.90th=[143655], 99.95th=[149947],
| 99.99th=[154141]
bw ( KiB/s): min=28000, max=66432, per=100.00%, avg=36152.40, stdev=1453.62, samples=720
iops : min= 1750, max= 4152, avg=2259.52, stdev=90.85, samples=720
lat (nsec) : 1000=0.02%
lat (usec) : 2=0.23%, 4=0.01%, 10=0.10%, 20=3.94%, 50=27.04%
lat (usec) : 100=1.63%, 250=0.77%, 500=0.11%, 750=0.01%, 1000=3.74%
lat (msec) : 2=16.01%, 4=32.22%, 10=3.77%, 20=0.50%, 50=9.20%
lat (msec) : 100=0.45%, 250=0.25%
cpu : usr=0.85%, sys=3.46%, ctx=494803, majf=0, minf=57
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=271711,271156,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=35.4MiB/s (37.1MB/s), 35.4MiB/s-35.4MiB/s (37.1MB/s-37.1MB/s), io=4245MiB (4452MB), run=120088-120088msec
WRITE: bw=35.3MiB/s (37.0MB/s), 35.3MiB/s-35.3MiB/s (37.0MB/s-37.0MB/s), io=4237MiB (4443MB), run=120088-120088msec
PRE-SNAPSHOT_RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 5842/4145152 216704/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 27738112 209715199
PRE-SNAPSHOT_RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-2: (groupid=0, jobs=3): err= 0: pid=5649: Fri Oct 14 18:09:53 2022
read: IOPS=2324, BW=36.3MiB/s (38.1MB/s)(4359MiB/120035msec)
slat (usec): min=4, max=154607, avg=74.75, stdev=2558.81
clat (nsec): min=800, max=175991k, avg=4069864.27, stdev=11277038.81
lat (usec): min=9, max=176006, avg=4144.78, stdev=11553.80
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 30],
| 30.00th=[ 1500], 40.00th=[ 1663], 50.00th=[ 1745], 60.00th=[ 1844],
| 70.00th=[ 1958], 80.00th=[ 2114], 90.00th=[ 2704], 95.00th=[ 32900],
| 99.00th=[ 43254], 99.50th=[ 50594], 99.90th=[139461], 99.95th=[145753],
| 99.99th=[152044]
bw ( KiB/s): min=26880, max=61280, per=100.00%, avg=37197.92, stdev=1524.93, samples=717
iops : min= 1680, max= 3830, avg=2324.87, stdev=95.31, samples=717
write: IOPS=2324, BW=36.3MiB/s (38.1MB/s)(4360MiB/120035msec); 0 zone resets
slat (usec): min=4, max=154559, avg=71.95, stdev=2478.54
clat (usec): min=712, max=164541, avg=6104.23, stdev=13467.46
lat (usec): min=724, max=164554, avg=6176.34, stdev=13679.15
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 865], 10.00th=[ 898], 20.00th=[ 955],
| 30.00th=[ 1004], 40.00th=[ 1074], 50.00th=[ 1172], 60.00th=[ 1663],
| 70.00th=[ 2245], 80.00th=[ 2638], 90.00th=[ 31589], 95.00th=[ 35914],
| 99.00th=[ 48497], 99.50th=[ 52691], 99.90th=[141558], 99.95th=[147850],
| 99.99th=[154141]
bw ( KiB/s): min=27744, max=59936, per=100.00%, avg=37211.72, stdev=1482.63, samples=717
iops : min= 1734, max= 3746, avg=2325.73, stdev=92.66, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=1.05%, 50=12.30%
lat (usec) : 100=0.76%, 250=0.25%, 500=0.01%, 750=0.01%, 1000=14.58%
lat (msec) : 2=38.65%, 4=21.06%, 10=1.54%, 20=0.22%, 50=8.90%
lat (msec) : 100=0.33%, 250=0.31%
cpu : usr=0.91%, sys=3.58%, ctx=489414, majf=0, minf=53
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=278972,279069,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=4359MiB (4571MB), run=120035-120035msec
WRITE: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=4360MiB (4572MB), run=120035-120035msec
PRE-SNAPSHOT_RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 6215/4145152 297471/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 38076288 209715199
PRE-SNAPSHOT_RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-3: (groupid=0, jobs=3): err= 0: pid=5676: Fri Oct 14 18:11:53 2022
read: IOPS=2437, BW=38.1MiB/s (39.9MB/s)(4570MiB/120002msec)
slat (usec): min=4, max=104382, avg=42.15, stdev=1294.67
clat (nsec): min=900, max=106010k, avg=4627223.21, stdev=10188097.09
lat (usec): min=11, max=106377, avg=4669.56, stdev=10263.39
clat percentiles (usec):
| 1.00th=[ 22], 5.00th=[ 26], 10.00th=[ 39], 20.00th=[ 1565],
| 30.00th=[ 1647], 40.00th=[ 1713], 50.00th=[ 1778], 60.00th=[ 1860],
| 70.00th=[ 1958], 80.00th=[ 2114], 90.00th=[ 3294], 95.00th=[ 33817],
| 99.00th=[ 44303], 99.50th=[ 49546], 99.90th=[ 87557], 99.95th=[ 93848],
| 99.99th=[103285]
bw ( KiB/s): min=31136, max=59232, per=100.00%, avg=39008.54, stdev=1046.77, samples=717
iops : min= 1946, max= 3702, avg=2438.03, stdev=65.42, samples=717
write: IOPS=2431, BW=38.0MiB/s (39.8MB/s)(4559MiB/120002msec); 0 zone resets
slat (usec): min=5, max=104446, avg=46.81, stdev=1432.73
clat (usec): min=727, max=108199, avg=5140.33, stdev=11212.37
lat (usec): min=738, max=108209, avg=5187.34, stdev=11295.15
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 857], 10.00th=[ 881], 20.00th=[ 930],
| 30.00th=[ 971], 40.00th=[ 1012], 50.00th=[ 1074], 60.00th=[ 1172],
| 70.00th=[ 1418], 80.00th=[ 2311], 90.00th=[ 25297], 95.00th=[ 34866],
| 99.00th=[ 45876], 99.50th=[ 50070], 99.90th=[ 86508], 99.95th=[ 93848],
| 99.99th=[101188]
bw ( KiB/s): min=31072, max=59584, per=100.00%, avg=38929.81, stdev=1067.52, samples=717
iops : min= 1942, max= 3724, avg=2433.11, stdev=66.72, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.29%, 50=5.16%
lat (usec) : 100=0.40%, 250=0.18%, 500=0.01%, 750=0.01%, 1000=18.49%
lat (msec) : 2=49.45%, 4=14.55%, 10=1.58%, 20=0.45%, 50=8.94%
lat (msec) : 100=0.48%, 250=0.01%
cpu : usr=0.96%, sys=3.82%, ctx=503237, majf=0, minf=58
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=292452,291786,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=38.1MiB/s (39.9MB/s), 38.1MiB/s-38.1MiB/s (39.9MB/s-39.9MB/s), io=4570MiB (4792MB), run=120002-120002msec
WRITE: bw=38.0MiB/s (39.8MB/s), 38.0MiB/s-38.0MiB/s (39.8MB/s-39.8MB/s), io=4559MiB (4781MB), run=120002-120002msec
PRE-SNAPSHOT_RUN 3/3 finished.
Logical volume "thinSnap1" created.
Logical volume "thinSnap1" successfully removed
POST SNAPSHOT-DELTE RUNS
==================
Running POST-SNAPSHOT-DELETE-RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6374/4145152 332516/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 42562048 209715199
POST-SNAPSHOT-DELETE-RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-1: (groupid=0, jobs=3): err= 0: pid=5727: Fri Oct 14 18:13:54 2022
read: IOPS=1297, BW=20.3MiB/s (21.3MB/s)(2434MiB/120045msec)
slat (usec): min=4, max=192261, avg=153.56, stdev=4494.34
clat (nsec): min=1000, max=200064k, avg=4977125.44, stdev=12906763.26
lat (usec): min=19, max=200077, avg=5130.90, stdev=13664.53
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 50], 10.00th=[ 1532], 20.00th=[ 1631],
| 30.00th=[ 1696], 40.00th=[ 1745], 50.00th=[ 1811], 60.00th=[ 1876],
| 70.00th=[ 1975], 80.00th=[ 2147], 90.00th=[ 3294], 95.00th=[ 32637],
| 99.00th=[ 42206], 99.50th=[ 49021], 99.90th=[158335], 99.95th=[168821],
| 99.99th=[191890]
bw ( KiB/s): min=12736, max=45568, per=100.00%, avg=20785.81, stdev=1315.83, samples=717
iops : min= 796, max= 2848, avg=1299.11, stdev=82.24, samples=717
write: IOPS=1299, BW=20.3MiB/s (21.3MB/s)(2437MiB/120045msec); 0 zone resets
slat (usec): min=5, max=192735, avg=167.04, stdev=4736.97
clat (usec): min=711, max=210474, avg=13175.19, stdev=20940.19
lat (usec): min=726, max=210485, avg=13342.44, stdev=21457.13
clat percentiles (usec):
| 1.00th=[ 848], 5.00th=[ 914], 10.00th=[ 996], 20.00th=[ 2278],
| 30.00th=[ 4113], 40.00th=[ 4293], 50.00th=[ 4555], 60.00th=[ 4883],
| 70.00th=[ 5997], 80.00th=[ 32637], 90.00th=[ 37487], 95.00th=[ 41157],
| 99.00th=[132645], 99.50th=[156238], 99.90th=[191890], 99.95th=[196084],
| 99.99th=[202376]
bw ( KiB/s): min=13664, max=43136, per=100.00%, avg=20814.86, stdev=1243.26, samples=717
iops : min= 854, max= 2696, avg=1300.93, stdev=77.70, samples=717
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.11%, 50=2.38%
lat (usec) : 100=0.27%, 250=0.09%, 500=0.02%, 750=0.01%, 1000=5.02%
lat (msec) : 2=36.99%, 4=13.48%, 10=24.54%, 20=0.72%, 50=15.17%
lat (msec) : 100=0.50%, 250=0.71%
cpu : usr=0.57%, sys=2.23%, ctx=293635, majf=0, minf=54
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=155796,155962,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=2434MiB (2553MB), run=120045-120045msec
WRITE: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=2437MiB (2555MB), run=120045-120045msec
POST-SNAPSHOT-DELETE-RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6420/4145152 341583/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 43722624 209715199
POST-SNAPSHOT-DELETE-RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-2: (groupid=0, jobs=3): err= 0: pid=5761: Fri Oct 14 18:15:54 2022
read: IOPS=1599, BW=25.0MiB/s (26.2MB/s)(3000MiB/120022msec)
slat (usec): min=4, max=173921, avg=121.05, stdev=3689.11
clat (nsec): min=1000, max=175563k, avg=5102987.92, stdev=12462800.54
lat (usec): min=22, max=176770, avg=5224.23, stdev=12985.10
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 1483], 10.00th=[ 1565], 20.00th=[ 1647],
| 30.00th=[ 1696], 40.00th=[ 1762], 50.00th=[ 1827], 60.00th=[ 1893],
| 70.00th=[ 1975], 80.00th=[ 2147], 90.00th=[ 3556], 95.00th=[ 33162],
| 99.00th=[ 43779], 99.50th=[ 50070], 99.90th=[147850], 99.95th=[152044],
| 99.99th=[166724]
bw ( KiB/s): min=17088, max=40480, per=100.00%, avg=25617.27, stdev=1308.97, samples=717
iops : min= 1068, max= 2530, avg=1601.08, stdev=81.81, samples=717
write: IOPS=1599, BW=25.0MiB/s (26.2MB/s)(3000MiB/120022msec); 0 zone resets
slat (usec): min=5, max=174306, avg=118.53, stdev=3607.88
clat (usec): min=639, max=202877, avg=9654.89, stdev=17492.21
lat (usec): min=734, max=202901, avg=9773.61, stdev=17844.74
clat percentiles (usec):
| 1.00th=[ 824], 5.00th=[ 873], 10.00th=[ 906], 20.00th=[ 971],
| 30.00th=[ 1045], 40.00th=[ 1205], 50.00th=[ 3687], 60.00th=[ 4293],
| 70.00th=[ 4752], 80.00th=[ 7832], 90.00th=[ 35390], 95.00th=[ 39060],
| 99.00th=[ 51643], 99.50th=[139461], 99.90th=[156238], 99.95th=[160433],
| 99.99th=[177210]
bw ( KiB/s): min=18208, max=40384, per=100.00%, avg=25620.22, stdev=1273.21, samples=717
iops : min= 1138, max= 2524, avg=1601.26, stdev=79.57, samples=717
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.06%, 50=1.42%
lat (usec) : 100=0.15%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=12.40%
lat (msec) : 2=44.69%, 4=12.42%, 10=14.76%, 20=0.43%, 50=12.71%
lat (msec) : 100=0.33%, 250=0.57%
cpu : usr=0.76%, sys=2.62%, ctx=353621, majf=0, minf=54
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=191996,192030,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=25.0MiB/s (26.2MB/s), 25.0MiB/s-25.0MiB/s (26.2MB/s-26.2MB/s), io=3000MiB (3146MB), run=120022-120022msec
WRITE: bw=25.0MiB/s (26.2MB/s), 25.0MiB/s-25.0MiB/s (26.2MB/s-26.2MB/s), io=3000MiB (3146MB), run=120022-120022msec
POST-SNAPSHOT-DELETE-RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6442/4145152 348165/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ext4.ThinLV: 0 209715200 thin 44565120 209715199
POST-SNAPSHOT-DELETE-RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-3: (groupid=0, jobs=3): err= 0: pid=5789: Fri Oct 14 18:17:54 2022
read: IOPS=1927, BW=30.1MiB/s (31.6MB/s)(3614MiB/120011msec)
slat (usec): min=4, max=132713, avg=78.46, stdev=2380.78
clat (nsec): min=800, max=148982k, avg=5172373.48, stdev=11432105.19
lat (usec): min=20, max=148993, avg=5251.00, stdev=11662.55
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 1516], 10.00th=[ 1582], 20.00th=[ 1647],
| 30.00th=[ 1713], 40.00th=[ 1762], 50.00th=[ 1827], 60.00th=[ 1893],
| 70.00th=[ 1991], 80.00th=[ 2180], 90.00th=[ 4424], 95.00th=[ 33817],
| 99.00th=[ 45351], 99.50th=[ 53740], 99.90th=[106431], 99.95th=[129500],
| 99.99th=[141558]
bw ( KiB/s): min=21600, max=48672, per=100.00%, avg=30852.55, stdev=1193.78, samples=717
iops : min= 1350, max= 3042, avg=1928.28, stdev=74.61, samples=717
write: IOPS=1924, BW=30.1MiB/s (31.5MB/s)(3609MiB/120011msec); 0 zone resets
slat (usec): min=5, max=132809, avg=70.33, stdev=2180.56
clat (usec): min=727, max=174899, avg=7136.89, stdev=13919.70
lat (usec): min=741, max=174914, avg=7207.37, stdev=14088.78
clat percentiles (usec):
| 1.00th=[ 824], 5.00th=[ 865], 10.00th=[ 898], 20.00th=[ 947],
| 30.00th=[ 996], 40.00th=[ 1057], 50.00th=[ 1139], 60.00th=[ 1319],
| 70.00th=[ 4015], 80.00th=[ 4817], 90.00th=[ 33424], 95.00th=[ 37487],
| 99.00th=[ 48497], 99.50th=[ 63177], 99.90th=[110625], 99.95th=[135267],
| 99.99th=[147850]
bw ( KiB/s): min=22400, max=46880, per=100.00%, avg=30810.11, stdev=1160.81, samples=717
iops : min= 1400, max= 2930, avg=1925.63, stdev=72.55, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.77%
lat (usec) : 100=0.06%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=15.71%
lat (msec) : 2=51.14%, 4=12.13%, 10=8.11%, 20=0.28%, 50=11.05%
lat (msec) : 100=0.47%, 250=0.22%
cpu : usr=0.82%, sys=3.10%, ctx=413673, majf=0, minf=52
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=231306,230998,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=30.1MiB/s (31.6MB/s), 30.1MiB/s-30.1MiB/s (31.6MB/s-31.6MB/s), io=3614MiB (3790MB), run=120011-120011msec
WRITE: bw=30.1MiB/s (31.5MB/s), 30.1MiB/s-30.1MiB/s (31.5MB/s-31.5MB/s), io=3609MiB (3785MB), run=120011-120011msec
POST-SNAPSHOT-DELETE-RUN 3/3 finished.
Raw
LVM_Snapshot_performance
Collected performance results for thin lvm volumes along with snapshot (Having snapshot and immediatly deleting snapshot without writing to original volume)
Ran FIO test and collected performance results on following scenarios
Test1:
1. Created thin LVM volume
2. Run FIO 3 times on thin LVM volume
3. Created thin snapshot and immediatly deleted the thin snapshot
4. Run again FIO 3 times on thin LVM volume
Note: Attached results to raw_perf.txt which contains output of `dmsetup status` & `dmsetup table`
Test2:
1. Created thin LVM volume and formated with XFS filesystem
2. Mounted XFS filesystem
2. Run FIO 3 times on file in XFS.
3. Created thin snapshot and immediatly deleted the thin snapshot
4. Run again FIO 3 times on file in XFS
Note: Attached results to xfs_perf.txt which contains output of `dmsetup status` & `dmsetup table`
Test3:
1. Created thin LVM volume and formated with Ext4 filesystem
2. Mounted Ext4 filesystem
2. Run FIO 3 times on file in Ext4.
3. Created thin snapshot and immediatly deleted the thin snapshot
4. Run again FIO 3 times on file in Ext4.
Note: Attached results to ext4_perf.txt which contains output of `dmsetup status` & `dmsetup table`
FIO command: fio --filename=/dev/ThinVolGrp/ThinLV --name=<name> --time_based --group_reporting --ioengine=libaio --direct=1 --size=20G --runtime=120 --rw=randrw --rwmixread=50 --bs=16K --iodepth=8 --numjobs=3 --randrepeat=0 --randseed=$(date +%s)
Raw
raw_perf.txt
Running PRE-SNAPSHOT_RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4843/4145152 0/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 0 -
PRE-SNAPSHOT_RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-1: (groupid=0, jobs=3): err= 0: pid=4443: Fri Oct 14 17:34:15 2022
read: IOPS=2361, BW=36.9MiB/s (38.7MB/s)(4428MiB/120004msec)
slat (usec): min=3, max=158812, avg=60.84, stdev=2268.29
clat (nsec): min=800, max=178835k, avg=1994654.21, stdev=8008847.02
lat (usec): min=8, max=178850, avg=2055.69, stdev=8320.50
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 23],
| 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 41],
| 70.00th=[ 1663], 80.00th=[ 1975], 90.00th=[ 2573], 95.00th=[ 3851],
| 99.00th=[ 37487], 99.50th=[ 42730], 99.90th=[111674], 99.95th=[147850],
| 99.99th=[156238]
bw ( KiB/s): min=26528, max=77600, per=100.00%, avg=37842.21, stdev=1752.43, samples=717
iops : min= 1658, max= 4850, avg=2365.14, stdev=109.53, samples=717
write: IOPS=2366, BW=37.0MiB/s (38.8MB/s)(4437MiB/120004msec); 0 zone resets
slat (usec): min=3, max=158496, avg=66.54, stdev=2433.97
clat (usec): min=733, max=186324, avg=8020.69, stdev=14728.17
lat (usec): min=745, max=186334, avg=8087.42, stdev=14917.14
clat percentiles (usec):
| 1.00th=[ 848], 5.00th=[ 930], 10.00th=[ 996], 20.00th=[ 1172],
| 30.00th=[ 2089], 40.00th=[ 2311], 50.00th=[ 2474], 60.00th=[ 2671],
| 70.00th=[ 2933], 80.00th=[ 4146], 90.00th=[ 34341], 95.00th=[ 38011],
| 99.00th=[ 50070], 99.50th=[ 60556], 99.90th=[147850], 99.95th=[152044],
| 99.99th=[160433]
bw ( KiB/s): min=27744, max=81152, per=100.00%, avg=37922.14, stdev=1659.52, samples=717
iops : min= 1734, max= 5072, avg=2370.13, stdev=103.72, samples=717
lat (nsec) : 1000=0.02%
lat (usec) : 2=0.21%, 4=0.01%, 10=0.12%, 20=4.04%, 50=27.05%
lat (usec) : 100=1.38%, 250=0.68%, 500=0.07%, 750=0.01%, 1000=5.03%
lat (msec) : 2=16.00%, 4=32.72%, 10=2.86%, 20=0.40%, 50=8.78%
lat (msec) : 100=0.38%, 250=0.24%
cpu : usr=0.94%, sys=3.39%, ctx=513116, majf=0, minf=59
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=283397,283973,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=4428MiB (4643MB), run=120004-120004msec
WRITE: bw=37.0MiB/s (38.8MB/s), 37.0MiB/s-37.0MiB/s (38.8MB/s-38.8MB/s), io=4437MiB (4653MB), run=120004-120004msec
PRE-SNAPSHOT_RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 5723/4145152 191509/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 24513152 41943039
PRE-SNAPSHOT_RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-2: (groupid=0, jobs=3): err= 0: pid=4478: Fri Oct 14 17:36:15 2022
read: IOPS=2324, BW=36.3MiB/s (38.1MB/s)(4360MiB/120031msec)
slat (usec): min=3, max=151714, avg=79.81, stdev=2626.23
clat (nsec): min=800, max=179134k, avg=4111593.30, stdev=11044939.49
lat (usec): min=10, max=179144, avg=4191.57, stdev=11341.38
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 31],
| 30.00th=[ 1516], 40.00th=[ 1680], 50.00th=[ 1778], 60.00th=[ 1876],
| 70.00th=[ 1991], 80.00th=[ 2212], 90.00th=[ 2966], 95.00th=[ 33162],
| 99.00th=[ 43779], 99.50th=[ 50070], 99.90th=[135267], 99.95th=[145753],
| 99.99th=[152044]
bw ( KiB/s): min=28096, max=59264, per=100.00%, avg=37239.43, stdev=1513.94, samples=717
iops : min= 1756, max= 3704, avg=2327.46, stdev=94.62, samples=717
write: IOPS=2323, BW=36.3MiB/s (38.1MB/s)(4357MiB/120031msec); 0 zone resets
slat (usec): min=3, max=146126, avg=65.93, stdev=2324.69
clat (usec): min=714, max=181401, avg=6066.06, stdev=13643.28
lat (usec): min=725, max=181411, avg=6132.16, stdev=13827.74
clat percentiles (usec):
| 1.00th=[ 807], 5.00th=[ 857], 10.00th=[ 889], 20.00th=[ 938],
| 30.00th=[ 988], 40.00th=[ 1057], 50.00th=[ 1139], 60.00th=[ 1483],
| 70.00th=[ 2212], 80.00th=[ 2540], 90.00th=[ 31327], 95.00th=[ 36439],
| 99.00th=[ 49021], 99.50th=[ 63177], 99.90th=[143655], 99.95th=[149947],
| 99.99th=[166724]
bw ( KiB/s): min=28192, max=59072, per=100.00%, avg=37216.80, stdev=1515.72, samples=717
iops : min= 1762, max= 3692, avg=2326.05, stdev=94.73, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=1.03%, 50=11.85%
lat (usec) : 100=0.78%, 250=0.32%, 500=0.02%, 750=0.01%, 1000=15.91%
lat (msec) : 2=36.80%, 4=21.94%, 10=1.52%, 20=0.25%, 50=8.85%
lat (msec) : 100=0.37%, 250=0.30%
cpu : usr=0.87%, sys=3.58%, ctx=485031, majf=0, minf=51
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=279059,278865,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=4360MiB (4572MB), run=120031-120031msec
WRITE: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=4357MiB (4569MB), run=120031-120031msec
PRE-SNAPSHOT_RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 6084/4145152 270107/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 34573696 41943039
PRE-SNAPSHOT_RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-3: (groupid=0, jobs=3): err= 0: pid=4532: Fri Oct 14 17:38:15 2022
read: IOPS=2441, BW=38.2MiB/s (40.0MB/s)(4579MiB/120026msec)
slat (usec): min=3, max=117475, avg=42.92, stdev=1334.64
clat (nsec): min=900, max=118978k, avg=4717521.34, stdev=10177151.88
lat (usec): min=11, max=119772, avg=4760.64, stdev=10257.04
clat percentiles (usec):
| 1.00th=[ 22], 5.00th=[ 27], 10.00th=[ 43], 20.00th=[ 1598],
| 30.00th=[ 1680], 40.00th=[ 1762], 50.00th=[ 1827], 60.00th=[ 1909],
| 70.00th=[ 2024], 80.00th=[ 2245], 90.00th=[ 4359], 95.00th=[ 33424],
| 99.00th=[ 45351], 99.50th=[ 49546], 99.90th=[ 88605], 99.95th=[ 94897],
| 99.99th=[105382]
bw ( KiB/s): min=32064, max=61088, per=100.00%, avg=39084.72, stdev=1022.66, samples=717
iops : min= 2004, max= 3818, avg=2442.79, stdev=63.92, samples=717
write: IOPS=2441, BW=38.2MiB/s (40.0MB/s)(4579MiB/120026msec); 0 zone resets
slat (usec): min=4, max=117104, avg=41.51, stdev=1254.89
clat (usec): min=729, max=119763, avg=5023.34, stdev=10972.72
lat (usec): min=740, max=119774, avg=5065.04, stdev=11037.05
clat percentiles (usec):
| 1.00th=[ 824], 5.00th=[ 865], 10.00th=[ 898], 20.00th=[ 947],
| 30.00th=[ 988], 40.00th=[ 1037], 50.00th=[ 1090], 60.00th=[ 1188],
| 70.00th=[ 1385], 80.00th=[ 2311], 90.00th=[ 25560], 95.00th=[ 34341],
| 99.00th=[ 45876], 99.50th=[ 49546], 99.90th=[ 82314], 99.95th=[ 90702],
| 99.99th=[102237]
bw ( KiB/s): min=32640, max=61856, per=100.00%, avg=39090.08, stdev=1033.67, samples=717
iops : min= 2040, max= 3866, avg=2443.13, stdev=64.60, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.29%, 50=4.91%
lat (usec) : 100=0.48%, 250=0.20%, 500=0.01%, 750=0.01%, 1000=16.15%
lat (msec) : 2=49.58%, 4=16.99%, 10=1.55%, 20=0.37%, 50=9.01%
lat (msec) : 100=0.42%, 250=0.02%
cpu : usr=0.94%, sys=3.77%, ctx=499159, majf=0, minf=50
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=293078,293087,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=38.2MiB/s (40.0MB/s), 38.2MiB/s-38.2MiB/s (40.0MB/s-40.0MB/s), io=4579MiB (4802MB), run=120026-120026msec
WRITE: bw=38.2MiB/s (40.0MB/s), 38.2MiB/s-38.2MiB/s (40.0MB/s-40.0MB/s), io=4579MiB (4802MB), run=120026-120026msec
PRE-SNAPSHOT_RUN 3/3 finished.
============================ Creating Snapshot =======================================
Logical volume "thinSnap1" created. ## Snapshot Created
Logical volume "thinSnap1" successfully removed #### Snapshot Deleted
POST SNAPSHOT-DELTE RUNS
==================
Running POST-SNAPSHOT-DELETE-RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6249/4145152 304312/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 38951936 41943039
============================================================================================
POST-SNAPSHOT-DELETE-RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-1: (groupid=0, jobs=3): err= 0: pid=4611: Fri Oct 14 17:40:16 2022
read: IOPS=1306, BW=20.4MiB/s (21.4MB/s)(2450MiB/120030msec)
slat (usec): min=4, max=186346, avg=164.81, stdev=4663.33
clat (nsec): min=1000, max=204136k, avg=4854057.41, stdev=12521949.61
lat (usec): min=18, max=204219, avg=5019.06, stdev=13381.37
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 52], 10.00th=[ 1532], 20.00th=[ 1647],
| 30.00th=[ 1713], 40.00th=[ 1778], 50.00th=[ 1844], 60.00th=[ 1926],
| 70.00th=[ 2040], 80.00th=[ 2245], 90.00th=[ 4293], 95.00th=[ 31065],
| 99.00th=[ 42730], 99.50th=[ 48497], 99.90th=[162530], 99.95th=[183501],
| 99.99th=[196084]
bw ( KiB/s): min=12576, max=41344, per=99.96%, avg=20895.87, stdev=1336.31, samples=717
iops : min= 786, max= 2584, avg=1305.99, stdev=83.52, samples=717
write: IOPS=1299, BW=20.3MiB/s (21.3MB/s)(2437MiB/120030msec); 0 zone resets
slat (usec): min=4, max=193381, avg=148.49, stdev=4409.36
clat (usec): min=742, max=227250, avg=13271.78, stdev=21070.96
lat (usec): min=754, max=227269, avg=13420.47, stdev=21542.26
clat percentiles (usec):
| 1.00th=[ 857], 5.00th=[ 930], 10.00th=[ 1020], 20.00th=[ 2343],
| 30.00th=[ 4178], 40.00th=[ 4424], 50.00th=[ 4686], 60.00th=[ 5145],
| 70.00th=[ 7111], 80.00th=[ 31065], 90.00th=[ 36439], 95.00th=[ 41681],
| 99.00th=[141558], 99.50th=[160433], 99.90th=[193987], 99.95th=[200279],
| 99.99th=[202376]
bw ( KiB/s): min=13824, max=42208, per=99.96%, avg=20782.73, stdev=1232.69, samples=717
iops : min= 864, max= 2638, avg=1298.92, stdev=77.04, samples=717
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.10%, 50=2.39%
lat (usec) : 100=0.26%, 250=0.09%, 500=0.02%, 750=0.01%, 1000=4.49%
lat (msec) : 2=35.00%, 4=15.13%, 10=25.00%, 20=1.36%, 50=14.97%
lat (msec) : 100=0.51%, 250=0.70%
cpu : usr=0.62%, sys=2.14%, ctx=294543, majf=0, minf=51
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=156813,155963,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=2450MiB (2569MB), run=120030-120030msec
WRITE: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=2437MiB (2555MB), run=120030-120030msec
POST-SNAPSHOT-DELETE-RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6283/4145152 313237/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 40094336 41943039
POST-SNAPSHOT-DELETE-RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-2: (groupid=0, jobs=3): err= 0: pid=4639: Fri Oct 14 17:42:16 2022
read: IOPS=1590, BW=24.8MiB/s (26.1MB/s)(2983MiB/120023msec)
slat (usec): min=4, max=163700, avg=117.58, stdev=3561.63
clat (nsec): min=1100, max=440844k, avg=5129512.51, stdev=12478589.22
lat (usec): min=16, max=440853, avg=5247.26, stdev=12964.36
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 1467], 10.00th=[ 1549], 20.00th=[ 1647],
| 30.00th=[ 1713], 40.00th=[ 1762], 50.00th=[ 1844], 60.00th=[ 1909],
| 70.00th=[ 2024], 80.00th=[ 2278], 90.00th=[ 4948], 95.00th=[ 32637],
| 99.00th=[ 43779], 99.50th=[ 50070], 99.90th=[145753], 99.95th=[152044],
| 99.99th=[164627]
bw ( KiB/s): min= 4960, max=37728, per=100.00%, avg=25483.92, stdev=1326.62, samples=717
iops : min= 310, max= 2358, avg=1592.74, stdev=82.91, samples=717
write: IOPS=1598, BW=25.0MiB/s (26.2MB/s)(2997MiB/120023msec); 0 zone resets
slat (usec): min=5, max=181000, avg=113.30, stdev=3482.64
clat (usec): min=702, max=445684, avg=9678.34, stdev=17689.42
lat (usec): min=715, max=445694, avg=9791.81, stdev=18025.24
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 857], 10.00th=[ 889], 20.00th=[ 947],
| 30.00th=[ 1020], 40.00th=[ 1172], 50.00th=[ 3720], 60.00th=[ 4293],
| 70.00th=[ 4817], 80.00th=[ 9241], 90.00th=[ 35390], 95.00th=[ 39060],
| 99.00th=[ 52691], 99.50th=[133694], 99.90th=[158335], 99.95th=[164627],
| 99.99th=[189793]
bw ( KiB/s): min= 4992, max=37952, per=100.00%, avg=25601.87, stdev=1302.33, samples=717
iops : min= 312, max= 2372, avg=1600.12, stdev=81.40, samples=717
lat (usec) : 2=0.01%, 20=0.06%, 50=1.45%, 100=0.12%, 250=0.05%
lat (usec) : 500=0.01%, 750=0.01%, 1000=13.98%
lat (msec) : 2=41.05%, 4=14.19%, 10=14.75%, 20=0.80%, 50=12.61%
lat (msec) : 100=0.35%, 250=0.57%, 500=0.01%
cpu : usr=0.72%, sys=2.52%, ctx=352201, majf=0, minf=57
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=190883,191823,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=24.8MiB/s (26.1MB/s), 24.8MiB/s-24.8MiB/s (26.1MB/s-26.1MB/s), io=2983MiB (3127MB), run=120023-120023msec
WRITE: bw=25.0MiB/s (26.2MB/s), 25.0MiB/s-25.0MiB/s (26.2MB/s-26.2MB/s), io=2997MiB (3143MB), run=120023-120023msec
POST-SNAPSHOT-DELETE-RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6314/4145152 319591/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-ThinLV: 0 209715200 thin 40907648 41943039
POST-SNAPSHOT-DELETE-RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-3: (groupid=0, jobs=3): err= 0: pid=4669: Fri Oct 14 17:44:17 2022
read: IOPS=1923, BW=30.1MiB/s (31.5MB/s)(3607MiB/120027msec)
slat (usec): min=4, max=126236, avg=81.22, stdev=2420.17
clat (nsec): min=1000, max=146227k, avg=5150510.08, stdev=11124194.52
lat (usec): min=23, max=146240, avg=5231.92, stdev=11373.22
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 1500], 10.00th=[ 1565], 20.00th=[ 1647],
| 30.00th=[ 1713], 40.00th=[ 1778], 50.00th=[ 1844], 60.00th=[ 1926],
| 70.00th=[ 2040], 80.00th=[ 2245], 90.00th=[ 5669], 95.00th=[ 33424],
| 99.00th=[ 43779], 99.50th=[ 52167], 99.90th=[106431], 99.95th=[109577],
| 99.99th=[137364]
bw ( KiB/s): min=21792, max=45792, per=100.00%, avg=30797.79, stdev=1160.56, samples=717
iops : min= 1362, max= 2862, avg=1924.86, stdev=72.54, samples=717
write: IOPS=1930, BW=30.2MiB/s (31.6MB/s)(3621MiB/120027msec); 0 zone resets
slat (usec): min=4, max=122693, avg=65.52, stdev=2062.20
clat (usec): min=735, max=153174, avg=7150.36, stdev=13855.58
lat (usec): min=744, max=158988, avg=7216.06, stdev=14003.17
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 857], 10.00th=[ 889], 20.00th=[ 930],
| 30.00th=[ 979], 40.00th=[ 1037], 50.00th=[ 1106], 60.00th=[ 1336],
| 70.00th=[ 4080], 80.00th=[ 4948], 90.00th=[ 32900], 95.00th=[ 37487],
| 99.00th=[ 49021], 99.50th=[ 66847], 99.90th=[110625], 99.95th=[130548],
| 99.99th=[145753]
bw ( KiB/s): min=22912, max=46112, per=100.00%, avg=30918.56, stdev=1162.02, samples=717
iops : min= 1432, max= 2882, avg=1932.41, stdev=72.63, samples=717
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.77%
lat (usec) : 100=0.06%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=17.25%
lat (msec) : 2=47.69%, 4=13.39%, 10=8.62%, 20=0.47%, 50=10.98%
lat (msec) : 100=0.50%, 250=0.22%
cpu : usr=0.81%, sys=3.07%, ctx=411926, majf=0, minf=51
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=230853,231724,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=30.1MiB/s (31.5MB/s), 30.1MiB/s-30.1MiB/s (31.5MB/s-31.5MB/s), io=3607MiB (3782MB), run=120027-120027msec
WRITE: bw=30.2MiB/s (31.6MB/s), 30.2MiB/s-30.2MiB/s (31.6MB/s-31.6MB/s), io=3621MiB (3797MB), run=120027-120027msec
POST-SNAPSHOT-DELETE-RUN 3/3 finished.
Raw
xfs_perf.txt
meta-data=/dev/ThinVolGrp/xfs.ThinLV isize=512 agcount=16, agsize=1638400 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=26214400, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=12800, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
Running PRE-SNAPSHOT_RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4848/4145152 819/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 104832 209715199
PRE-SNAPSHOT_RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-1: (groupid=0, jobs=3): err= 0: pid=4990: Fri Oct 14 17:50:16 2022
read: IOPS=2362, BW=36.9MiB/s (38.7MB/s)(4430MiB/120020msec)
slat (usec): min=3, max=198630, avg=68.34, stdev=2472.34
clat (nsec): min=800, max=200308k, avg=2027297.82, stdev=8244734.02
lat (usec): min=7, max=201909, avg=2095.83, stdev=8602.63
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 23],
| 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 32], 60.00th=[ 42],
| 70.00th=[ 1680], 80.00th=[ 1975], 90.00th=[ 2507], 95.00th=[ 3687],
| 99.00th=[ 37487], 99.50th=[ 42730], 99.90th=[131597], 99.95th=[145753],
| 99.99th=[154141]
bw ( KiB/s): min=22560, max=82074, per=100.00%, avg=37839.64, stdev=1831.73, samples=717
iops : min= 1410, max= 5128, avg=2364.97, stdev=114.46, samples=717
write: IOPS=2362, BW=36.9MiB/s (38.7MB/s)(4431MiB/120020msec); 0 zone resets
slat (usec): min=3, max=198615, avg=61.46, stdev=2251.89
clat (usec): min=706, max=203690, avg=7997.61, stdev=14643.06
lat (usec): min=719, max=203701, avg=8059.26, stdev=14804.36
clat percentiles (usec):
| 1.00th=[ 857], 5.00th=[ 938], 10.00th=[ 1012], 20.00th=[ 1172],
| 30.00th=[ 2073], 40.00th=[ 2311], 50.00th=[ 2474], 60.00th=[ 2638],
| 70.00th=[ 2900], 80.00th=[ 4228], 90.00th=[ 34341], 95.00th=[ 38011],
| 99.00th=[ 49546], 99.50th=[ 61080], 99.90th=[145753], 99.95th=[152044],
| 99.99th=[158335]
bw ( KiB/s): min=25376, max=81340, per=100.00%, avg=37855.31, stdev=1694.19, samples=717
iops : min= 1586, max= 5082, avg=2365.95, stdev=105.87, samples=717
lat (nsec) : 1000=0.02%
lat (usec) : 2=0.20%, 4=0.02%, 10=0.11%, 20=3.97%, 50=26.91%
lat (usec) : 100=1.58%, 250=0.68%, 500=0.07%, 750=0.01%, 1000=4.68%
lat (msec) : 2=16.53%, 4=32.69%, 10=2.72%, 20=0.35%, 50=8.86%
lat (msec) : 100=0.35%, 250=0.24%
cpu : usr=0.96%, sys=3.46%, ctx=515273, majf=0, minf=53
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=283493,283582,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=4430MiB (4645MB), run=120020-120020msec
WRITE: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=4431MiB (4646MB), run=120020-120020msec
PRE-SNAPSHOT_RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 5724/4145152 191830/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 24554240 209715199
PRE-SNAPSHOT_RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-2: (groupid=0, jobs=3): err= 0: pid=5077: Fri Oct 14 17:52:16 2022
read: IOPS=2323, BW=36.3MiB/s (38.1MB/s)(4358MiB/120007msec)
slat (usec): min=4, max=148920, avg=71.47, stdev=2476.14
clat (nsec): min=800, max=175424k, avg=4118481.41, stdev=11139666.64
lat (usec): min=10, max=175432, avg=4190.12, stdev=11405.01
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 31],
| 30.00th=[ 1516], 40.00th=[ 1680], 50.00th=[ 1778], 60.00th=[ 1876],
| 70.00th=[ 1991], 80.00th=[ 2180], 90.00th=[ 2966], 95.00th=[ 32637],
| 99.00th=[ 44827], 99.50th=[ 51119], 99.90th=[137364], 99.95th=[145753],
| 99.99th=[152044]
bw ( KiB/s): min=27072, max=65664, per=100.00%, avg=37189.36, stdev=1555.60, samples=717
iops : min= 1692, max= 4104, avg=2324.33, stdev=97.23, samples=717
write: IOPS=2327, BW=36.4MiB/s (38.1MB/s)(4365MiB/120007msec); 0 zone resets
slat (usec): min=3, max=148918, avg=74.41, stdev=2526.50
clat (usec): min=719, max=188562, avg=6049.29, stdev=13463.23
lat (usec): min=732, max=188575, avg=6123.86, stdev=13684.72
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 865], 10.00th=[ 906], 20.00th=[ 963],
| 30.00th=[ 1012], 40.00th=[ 1090], 50.00th=[ 1188], 60.00th=[ 1565],
| 70.00th=[ 2245], 80.00th=[ 2606], 90.00th=[ 31327], 95.00th=[ 35914],
| 99.00th=[ 49021], 99.50th=[ 57410], 99.90th=[143655], 99.95th=[147850],
| 99.99th=[154141]
bw ( KiB/s): min=27712, max=63552, per=100.00%, avg=37252.15, stdev=1530.14, samples=717
iops : min= 1732, max= 3972, avg=2328.26, stdev=95.63, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=1.08%, 50=12.03%
lat (usec) : 100=0.70%, 250=0.29%, 500=0.01%, 750=0.01%, 1000=13.72%
lat (msec) : 2=38.66%, 4=22.22%, 10=1.48%, 20=0.21%, 50=8.83%
lat (msec) : 100=0.42%, 250=0.29%
cpu : usr=0.95%, sys=3.52%, ctx=487498, majf=0, minf=49
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=278887,279349,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=4358MiB (4569MB), run=120007-120007msec
WRITE: bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=4365MiB (4577MB), run=120007-120007msec
PRE-SNAPSHOT_RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 6088/4145152 270805/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 34663040 209715199
PRE-SNAPSHOT_RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
PRE-SNAPSHOT_RUN-3: (groupid=0, jobs=3): err= 0: pid=5123: Fri Oct 14 17:54:16 2022
read: IOPS=2439, BW=38.1MiB/s (40.0MB/s)(4574MiB/120012msec)
slat (usec): min=3, max=102686, avg=39.76, stdev=1217.05
clat (nsec): min=900, max=117581k, avg=4699370.47, stdev=10179734.22
lat (usec): min=11, max=117594, avg=4739.32, stdev=10245.84
clat percentiles (usec):
| 1.00th=[ 22], 5.00th=[ 27], 10.00th=[ 39], 20.00th=[ 1565],
| 30.00th=[ 1663], 40.00th=[ 1729], 50.00th=[ 1795], 60.00th=[ 1876],
| 70.00th=[ 1991], 80.00th=[ 2212], 90.00th=[ 4178], 95.00th=[ 33817],
| 99.00th=[ 44303], 99.50th=[ 50070], 99.90th=[ 86508], 99.95th=[ 94897],
| 99.99th=[103285]
bw ( KiB/s): min=30944, max=59104, per=100.00%, avg=39061.15, stdev=1089.45, samples=717
iops : min= 1934, max= 3694, avg=2441.34, stdev=68.09, samples=717
write: IOPS=2440, BW=38.1MiB/s (40.0MB/s)(4576MiB/120012msec); 0 zone resets
slat (usec): min=4, max=102722, avg=43.87, stdev=1312.98
clat (usec): min=690, max=123477, avg=5050.62, stdev=11321.85
lat (usec): min=702, max=123490, avg=5094.67, stdev=11389.00
clat percentiles (usec):
| 1.00th=[ 799], 5.00th=[ 832], 10.00th=[ 865], 20.00th=[ 898],
| 30.00th=[ 938], 40.00th=[ 979], 50.00th=[ 1029], 60.00th=[ 1106],
| 70.00th=[ 1270], 80.00th=[ 2212], 90.00th=[ 25035], 95.00th=[ 34866],
| 99.00th=[ 47449], 99.50th=[ 51643], 99.90th=[ 86508], 99.95th=[ 92799],
| 99.99th=[104334]
bw ( KiB/s): min=31040, max=58144, per=100.00%, avg=39085.39, stdev=1012.78, samples=717
iops : min= 1940, max= 3634, avg=2442.84, stdev=63.30, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.29%, 50=5.08%
lat (usec) : 100=0.30%, 250=0.21%, 500=0.01%, 750=0.02%, 1000=22.44%
lat (msec) : 2=45.45%, 4=14.83%, 10=1.52%, 20=0.37%, 50=8.91%
lat (msec) : 100=0.53%, 250=0.02%
cpu : usr=0.95%, sys=3.61%, ctx=505877, majf=0, minf=51
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=292721,292889,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=38.1MiB/s (40.0MB/s), 38.1MiB/s-38.1MiB/s (40.0MB/s-40.0MB/s), io=4574MiB (4796MB), run=120012-120012msec
WRITE: bw=38.1MiB/s (40.0MB/s), 38.1MiB/s-38.1MiB/s (40.0MB/s-40.0MB/s), io=4576MiB (4799MB), run=120012-120012msec
PRE-SNAPSHOT_RUN 3/3 finished.
Logical volume "thinSnap1" created.
Logical volume "thinSnap1" successfully removed
POST SNAPSHOT-DELTE RUNS
==================
Running POST-SNAPSHOT-DELETE-RUN x 3 times
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6268/4145152 304880/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 39024640 209715199
POST-SNAPSHOT-DELETE-RUN-1: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-1: (groupid=0, jobs=3): err= 0: pid=5217: Fri Oct 14 17:56:17 2022
read: IOPS=1303, BW=20.4MiB/s (21.4MB/s)(2444MiB/120037msec)
slat (usec): min=4, max=196596, avg=156.92, stdev=4541.22
clat (nsec): min=900, max=221467k, avg=4939915.41, stdev=12953872.55
lat (usec): min=13, max=221486, avg=5097.05, stdev=13733.92
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 45], 10.00th=[ 1516], 20.00th=[ 1614],
| 30.00th=[ 1680], 40.00th=[ 1745], 50.00th=[ 1811], 60.00th=[ 1876],
| 70.00th=[ 1991], 80.00th=[ 2212], 90.00th=[ 3949], 95.00th=[ 32113],
| 99.00th=[ 43254], 99.50th=[ 49546], 99.90th=[162530], 99.95th=[183501],
| 99.99th=[196084]
bw ( KiB/s): min=13184, max=41312, per=100.00%, avg=20874.98, stdev=1266.46, samples=717
iops : min= 824, max= 2582, avg=1304.69, stdev=79.15, samples=717
write: IOPS=1299, BW=20.3MiB/s (21.3MB/s)(2438MiB/120037msec); 0 zone resets
slat (usec): min=5, max=219966, avg=156.60, stdev=4554.21
clat (usec): min=726, max=239248, avg=13194.71, stdev=21012.22
lat (usec): min=738, max=239264, avg=13351.52, stdev=21474.96
clat percentiles (usec):
| 1.00th=[ 848], 5.00th=[ 930], 10.00th=[ 1012], 20.00th=[ 2278],
| 30.00th=[ 4080], 40.00th=[ 4293], 50.00th=[ 4555], 60.00th=[ 4948],
| 70.00th=[ 6325], 80.00th=[ 31851], 90.00th=[ 36963], 95.00th=[ 41157],
| 99.00th=[121111], 99.50th=[158335], 99.90th=[191890], 99.95th=[193987],
| 99.99th=[219153]
bw ( KiB/s): min=13824, max=40960, per=100.00%, avg=20816.74, stdev=1232.49, samples=717
iops : min= 864, max= 2560, avg=1301.05, stdev=77.03, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.11%, 50=2.47%
lat (usec) : 100=0.22%, 250=0.07%, 500=0.01%, 750=0.01%, 1000=4.62%
lat (msec) : 2=36.79%, 4=14.10%, 10=24.44%, 20=0.91%, 50=14.94%
lat (msec) : 100=0.61%, 250=0.70%
cpu : usr=0.61%, sys=2.09%, ctx=295236, majf=0, minf=53
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=156429,156009,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=2444MiB (2563MB), run=120037-120037msec
WRITE: bw=20.3MiB/s (21.3MB/s), 20.3MiB/s-20.3MiB/s (21.3MB/s-21.3MB/s), io=2438MiB (2556MB), run=120037-120037msec
POST-SNAPSHOT-DELETE-RUN 1/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6296/4145152 313862/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 40174336 209715199
POST-SNAPSHOT-DELETE-RUN-2: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-2: (groupid=0, jobs=3): err= 0: pid=5258: Fri Oct 14 17:58:17 2022
read: IOPS=1600, BW=25.0MiB/s (26.2MB/s)(3003MiB/120039msec)
slat (usec): min=3, max=158823, avg=116.58, stdev=3513.10
clat (nsec): min=1000, max=180465k, avg=5089413.64, stdev=12355342.54
lat (usec): min=18, max=180479, avg=5206.19, stdev=12836.33
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 1467], 10.00th=[ 1532], 20.00th=[ 1614],
| 30.00th=[ 1680], 40.00th=[ 1745], 50.00th=[ 1795], 60.00th=[ 1876],
| 70.00th=[ 1975], 80.00th=[ 2180], 90.00th=[ 4752], 95.00th=[ 32900],
| 99.00th=[ 44303], 99.50th=[ 50070], 99.90th=[149947], 99.95th=[154141],
| 99.99th=[158335]
bw ( KiB/s): min=13760, max=42199, per=100.00%, avg=25641.87, stdev=1370.49, samples=717
iops : min= 860, max= 2636, avg=1602.61, stdev=85.65, samples=717
write: IOPS=1600, BW=25.0MiB/s (26.2MB/s)(3001MiB/120039msec); 0 zone resets
slat (usec): min=5, max=158947, avg=114.49, stdev=3505.78
clat (usec): min=720, max=282998, avg=9670.18, stdev=17575.11
lat (usec): min=735, max=283027, avg=9784.87, stdev=17906.00
clat percentiles (usec):
| 1.00th=[ 824], 5.00th=[ 865], 10.00th=[ 906], 20.00th=[ 971],
| 30.00th=[ 1057], 40.00th=[ 1221], 50.00th=[ 3720], 60.00th=[ 4293],
| 70.00th=[ 4752], 80.00th=[ 8586], 90.00th=[ 35390], 95.00th=[ 39060],
| 99.00th=[ 53216], 99.50th=[139461], 99.90th=[156238], 99.95th=[160433],
| 99.99th=[244319]
bw ( KiB/s): min=15552, max=41175, per=100.00%, avg=25635.85, stdev=1302.22, samples=717
iops : min= 972, max= 2572, avg=1602.23, stdev=81.38, samples=717
lat (usec) : 2=0.01%, 10=0.01%, 20=0.07%, 50=1.42%, 100=0.14%
lat (usec) : 250=0.06%, 500=0.01%, 750=0.01%, 1000=11.95%
lat (msec) : 2=45.06%, 4=12.38%, 10=14.77%, 20=0.61%, 50=12.56%
lat (msec) : 100=0.40%, 250=0.57%, 500=0.01%
cpu : usr=0.73%, sys=2.65%, ctx=352029, majf=0, minf=52
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=192176,192066,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=25.0MiB/s (26.2MB/s), 25.0MiB/s-25.0MiB/s (26.2MB/s-26.2MB/s), io=3003MiB (3149MB), run=120039-120039msec
WRITE: bw=25.0MiB/s (26.2MB/s), 25.0MiB/s-25.0MiB/s (26.2MB/s-26.2MB/s), io=3001MiB (3147MB), run=120039-120039msec
POST-SNAPSHOT-DELETE-RUN 2/3 finished.
dmsetup table
ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 253:2 1
dmsetup status
ThinVolGrp-ThinDataLV: 0 1006632960 linear
ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 3 6327/4145152 320428/7864320 - rw discard_passdown queue_if_no_space - 1024
ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear
ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear
ThinVolGrp-xfs.ThinLV: 0 209715200 thin 41014784 209715199
POST-SNAPSHOT-DELETE-RUN-3: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
...
fio-3.28
Starting 3 processes
POST-SNAPSHOT-DELETE-RUN-3: (groupid=0, jobs=3): err= 0: pid=5299: Fri Oct 14 18:00:18 2022
read: IOPS=1925, BW=30.1MiB/s (31.5MB/s)(3610MiB/120006msec)
slat (usec): min=4, max=160968, avg=77.99, stdev=2416.62
clat (nsec): min=900, max=190299k, avg=5107212.59, stdev=11258499.34
lat (usec): min=18, max=190311, avg=5185.36, stdev=11502.23
clat percentiles (usec):
| 1.00th=[ 30], 5.00th=[ 1483], 10.00th=[ 1549], 20.00th=[ 1614],
| 30.00th=[ 1680], 40.00th=[ 1745], 50.00th=[ 1795], 60.00th=[ 1876],
| 70.00th=[ 1975], 80.00th=[ 2180], 90.00th=[ 6063], 95.00th=[ 33162],
| 99.00th=[ 45351], 99.50th=[ 54789], 99.90th=[107480], 99.95th=[114820],
| 99.99th=[147850]
bw ( KiB/s): min=19840, max=49952, per=99.99%, avg=30803.82, stdev=1252.18, samples=717
iops : min= 1240, max= 3122, avg=1925.24, stdev=78.26, samples=717
write: IOPS=1923, BW=30.1MiB/s (31.5MB/s)(3606MiB/120006msec); 0 zone resets
slat (usec): min=5, max=180102, avg=72.52, stdev=2275.14
clat (usec): min=711, max=248423, avg=7211.43, stdev=14335.20
lat (usec): min=722, max=248436, avg=7284.11, stdev=14505.76
clat percentiles (usec):
| 1.00th=[ 807], 5.00th=[ 848], 10.00th=[ 881], 20.00th=[ 930],
| 30.00th=[ 971], 40.00th=[ 1029], 50.00th=[ 1106], 60.00th=[ 1319],
| 70.00th=[ 4015], 80.00th=[ 4883], 90.00th=[ 32637], 95.00th=[ 37487],
| 99.00th=[ 51643], 99.50th=[ 84411], 99.90th=[113771], 99.95th=[141558],
| 99.99th=[187696]
bw ( KiB/s): min=20096, max=51232, per=99.98%, avg=30767.00, stdev=1210.51, samples=717
iops : min= 1256, max= 3202, avg=1922.94, stdev=75.66, samples=717
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 20=0.03%, 50=0.78%, 100=0.06%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%, 1000=17.73%
lat (msec) : 2=49.54%, 4=11.19%, 10=8.52%, 20=0.56%, 50=10.69%
lat (msec) : 100=0.62%, 250=0.25%
cpu : usr=0.83%, sys=3.00%, ctx=413310, majf=0, minf=50
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=231064,230812,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=30.1MiB/s (31.5MB/s), 30.1MiB/s-30.1MiB/s (31.5MB/s-31.5MB/s), io=3610MiB (3786MB), run=120006-120006msec
WRITE: bw=30.1MiB/s (31.5MB/s), 30.1MiB/s-30.1MiB/s (31.5MB/s-31.5MB/s), io=3606MiB (3782MB), run=120006-120006msec
POST-SNAPSHOT-DELETE-RUN 3/3 finished.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment