Skip to content

Instantly share code, notes, and snippets.

@vschiavoni

vschiavoni/alpine-fio-musl, random writes

Last active April 29, 2020 20:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vschiavoni/dac91aea3d0f842f7b35c5db1b40c2d2 to your computer and use it in GitHub Desktop.
Save vschiavoni/dac91aea3d0f842f7b35c5db1b40c2d2 to your computer and use it in GitHub Desktop.
Download ZIP
tests with FIO on CASTOR
Raw
alpine-fio-musl, random writes
vschiavoni@castor-1:~/fio$ sudo docker build -t vschiavoni/alpine-fio-musl .
vschiavoni@castor-1:~/fio$ sudo docker run -ti vschiavoni/alpine-fio-musl /bin/sh
/ # fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --numjobs=2 --runtime=240 --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.19-48-g3966
Starting 2 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: (groupid=0, jobs=2): err= 0: pid=49: Wed Apr 22 20:32:14 2020
write: IOPS=442k, BW=1727MiB/s (1811MB/s)(1024MiB/593msec); 0 zone resets
slat (nsec): min=2036, max=68843, avg=3704.57, stdev=1098.36
clat (nsec): min=284, max=29436, avg=368.72, stdev=168.86
lat (nsec): min=2404, max=69500, avg=4143.20, stdev=1185.72
clat percentiles (nsec):
| 1.00th=[ 290], 5.00th=[ 294], 10.00th=[ 298], 20.00th=[ 314],
| 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 334], 60.00th=[ 342],
| 70.00th=[ 398], 80.00th=[ 442], 90.00th=[ 454], 95.00th=[ 462],
| 99.00th=[ 1004], 99.50th=[ 1144], 99.90th=[ 1224], 99.95th=[ 1256],
| 99.99th=[ 5792]
lat (nsec) : 500=98.36%, 750=0.58%, 1000=0.02%
lat (usec) : 2=1.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.27%, sys=88.39%, ctx=98, majf=0, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1727MiB/s (1811MB/s), 1727MiB/s-1727MiB/s (1811MB/s-1811MB/s), io=1024MiB (1074MB), run=593-593msec
/ #
Raw
alpine-fio-nvcache, random writes
139 vschiavoni@castor-1:~/fio$ sudo docker run --rm --device=/dev/pmem1 -ti vschiavoni/alpine-fio-nvcache /bin/sh
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 1
INFO: ------------------- z ]
INFO: ENABLE RECOVER = 1
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 195 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: PMEM IS NOT EMPTY
INFO: Starting recovery procedure...
]
INFO: -- Open @KL [FAILED]
INFO: -- Open aDZN
(]6
_<)SʠpA_gp!E
(a shitload of these..)
INFO: -- Open &11n42#DŅ(S;eu7#CX%CDyp;2I"l* [FAILED]
p61[QT+8 [FAILED]
INFO: -- Open L܋oatCȻWS؄#h͹l{,O,,=<7iذYm [FAILED]
INFO: -- Open [FAILED]
INFO: -- Open i~Zc$!i4k1GfQWAr3̯]naD&:<pWX
ZLڹP=Mqezvy[C
}MVN~39NVa+Cbq>CZ
({u
vW
җ [FAILED]
:!L*<ztgq8WK4PAQ#Vl$rtn%5CqsS/p+*4m*x}~㈶
Ҝ>\&/%Fux}S5;Up$c)]MPs[`Z&Ya>6ST}P [FAILED]
INFO: -- Open {@;K<گu_%p$hRϞh}^I
M٤m%oI!Z)' [FAILED]
INFO: -- Open ,.g>I79Sh5 [FAILED]
INFO: -- Open ugеć>Qx?oέ)V$qBRÂ >;b&G^%]߲%ѮwuUDžquHX&n'Ѳ|XN+
whʾ؟Ao40KjccaܒO*.u#xL'P^PIķa_Ccu
[FAILED]
INFO: -- Open ԰C9M@|d< w6k2l k}$RYʇaA,)ڬaި [FAILED]
ѐ4M%q^Yxp"?.ruԗOy!@7_6))v}f]~'2luFڽk^nnw [FAILED]
INFO: -- Open OOk@ [FAILED]
INFO: -- Open &k6
KMf^r$9ߕ
TityKP#AAruE1sZpl [FAILED]
oZ՟{?Zfu7K,&OBDR_WBkq=c[qT:s [FAILED]
INFO: -- Open ,w/g,tzCMw@2A4{\]$q^mUpe&P &YFsܔO>Þ
]#? [FAILED]
b#IRr2Z"n=Wc}gOHvoZ'֥uû)Cj`mANvl<TPj`F`q#@i"JVVud{8! P&pgi4,ɟU\tL[{>+]1tCb [FAILED]
INFO: -- Open _R [FAILED]
INFO: -- Open /i?mXqf
Ae!p<>M#ږ( )LP@2> [FAILED]SS^
INFO: -- Open ԅճj<wWf4❎؇zLi05a**)ZxZ1=`KUzn? [FAILED]
INFO: -- Open n`Cβ׺e.p%Tm|MhK3+:<%\}OHR6\йFD\^M3'$b7禟a+g
a06*
1
0SŰT{ dk`Xa̤p#h;;
|*Ǒq_#|iO [FAILED]
aƅgT+KVzPk`u@ېvATOAS@Wx#qaWb-6Kf"Rlj=}4 [FAILED]
INFO:
INFO: Flushing PMEM to disk...
INFO: -- PMEM flushed. --
INFO: Statistics on 5 entries :
INFO: -----> Flushed : 0 entries
INFO: -----> Ignored : 5000000 entries
INFO:
INFO: Continuing with a clean log.
INFO: -- Starting flushing thread --
Raw
core-dumping, alpine-fio-nvcache, random writes
vschiavoni@castor-1:~/fio$ sudo docker run --rm --device=/dev/pmem1 --env NVCACHE_ENABLE_RECOVER=false -ti vschiavoni/alpine-fio-nvcache /bin/sh
[sudo] password for vschiavoni:
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 1
INFO: -------------------
INFO: ENABLE RECOVER = 0
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 195 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: -- Starting flushing thread --
/ # fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --numjobs=2 --runtime=240 --group_reporting
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 1
INFO: -------------------
INFO: ENABLE RECOVER = 0
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 195 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: -- Starting flushing thread --
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.19-48-g3966
Starting 2 processes
randwrite: Laying out IO file (1 file / 512MiB)
randwrite: Laying out IO file (1 file / 512MiB)
Assertion failed: ret == 1 (helper_thread.c: submit_action: 95)
Aborted (core dumped)
Raw
debuggin coredump
#ioengine=null
io 140 fill: io_u 0x5575b6ad6c00: off=0x1ffff000,len=0x1000,ddir=1,file=null_seqwrite.0.0
io 140 prep: io_u 0x5575b6ad6c00: off=0x1ffff000,len=0x1000,ddir=1,file=null_seqwrite.0.0
io 140 queue: io_u 0x5575b6ad6c00: off=0x1ffff000,len=0x1000,ddir=1,file=null_seqwrite.0.0
io 140 complete: io_u 0x5575b6ad6c00: off=0x1ffff000,len=0x1000,ddir=1,file=null_seqwrite.0.0
file 140 put file null_seqwrite.0.0, ref=2
file 140 close files
file 140 put file null_seqwrite.0.0, ref=1
io 140 close ioengine null
io 140 free ioengine null
Assertion failed: ret == 1 (helper_thread.c: submit_action: 95)
Aborted (core dumped)
#ioengine=synch
file 186 put file null_seqwrite.0.0, ref=1
file 186 fd close null_seqwrite.0.0
io 186 close ioengine sync5KiB/s][w=233 IOPS][eta 00m:00s]
io 186 free ioengine sync
Assertion failed: ret == 1 (helper_thread.c: submit_action: 95)
Aborted (core dumped)
Raw
ioengine=null, sequential-writes (rw=writes)
sudo docker run --rm --device=/dev/pmem1 --env NVCACHE_ENABLE_RECOVER=false -ti vschiavoni/alpine-fio-nvcache /bin/sh
/ # fio --name=null_seqwrite --ioengine=null --iodepth=1 --rw=write --bs=4k --size=512M --numjobs=1 --runtime=30 --group_reporting
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 1
INFO: -------------------
INFO: ENABLE RECOVER = 0
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 195 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: -- Starting flushing thread --
randwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.19-48-g3966
Starting 1 process
Assertion failed: ret == 1 (helper_thread.c: submit_action: 95)
Aborted (core dumped)
Raw
ioengine=null, sequential-writes (rw=writes), libc 29.04.20
vschiavoni@castor-1:~/fio$ sudo docker run --rm --device=/dev/pmem1 --env NVCACHE_ENABLE_RECOVER=false -ti vschiavoni/alpine-fio-nvcache /bin/sh
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 10000
INFO: -------------------
INFO: ENABLE RECOVER = 0
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 1953 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: -- Starting flushing thread --
/ # fio --name=null_seqwrite --ioengine=null --iodepth=1 --rw=write --bs=4k --size=512M --numjobs=1 --runtime=30 --group_reporting
INFO: == Config ==
INFO: LOG SIZE = 5000000
INFO: -------------------
INFO: MAX BATCH SIZE = 120000
INFO: MIN BATCH SIZE = 10000
INFO: -------------------
INFO: ENABLE RECOVER = 0
INFO: FLUSH THREAD = 1
INFO: ============
INFO: Creating radix nodes.
radix_last_level = 4
INFO: --- Radix Cache ---
Cache size : 1953 MB
--- --- --- --- ---
INFO: Cache table initiated
-------------------------------------
INFO: -- Starting flushing thread --
null_seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.19-48-g3966
Starting 1 process
Assertion failed: ret == 1 (helper_thread.c: submit_action: 95)
Aborted (core dumped)
Raw
psync-sync baseline
/ # fio --name=null --ioengine=psync --iodepth=1 --rw=write --bs=4k --size=512M --numjobs=1 --runtime=30 --group_reporting
null: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.19-48-g3966
Starting 1 process
null: Laying out IO file (1 file / 512MiB)
null: (groupid=0, jobs=1): err= 0: pid=153: Thu Apr 23 08:18:51 2020
write: IOPS=370k, BW=1446MiB/s (1517MB/s)(512MiB/354msec); 0 zone resets
clat (nsec): min=1408, max=35692, avg=2443.77, stdev=829.26
lat (nsec): min=1461, max=36771, avg=2498.29, stdev=839.09
clat percentiles (nsec):
| 1.00th=[ 1752], 5.00th=[ 1768], 10.00th=[ 1832], 20.00th=[ 1880],
| 30.00th=[ 1944], 40.00th=[ 2064], 50.00th=[ 2448], 60.00th=[ 2512],
| 70.00th=[ 2576], 80.00th=[ 2608], 90.00th=[ 3216], 95.00th=[ 3856],
| 99.00th=[ 5280], 99.50th=[ 6048], 99.90th=[ 9152], 99.95th=[12352],
| 99.99th=[18560]
lat (usec) : 2=35.38%, 4=59.99%, 10=4.56%, 20=0.07%, 50=0.01%
cpu : usr=14.45%, sys=85.27%, ctx=30, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1446MiB/s (1517MB/s), 1446MiB/s-1446MiB/s (1517MB/s-1517MB/s), io=512MiB (537MB), run=354-354msec
####
/ # fio --name=null --ioengine=sync --iodepth=1 --rw=write --bs=4k --size=512M --numjobs=1 --runtime=30 --group_reporting
null: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.19-48-g3966
Starting 1 process
null: (groupid=0, jobs=1): err= 0: pid=196: Thu Apr 23 08:19:07 2020
write: IOPS=338k, BW=1320MiB/s (1384MB/s)(512MiB/388msec); 0 zone resets
clat (nsec): min=1551, max=127685, avg=2218.83, stdev=770.34
lat (nsec): min=1584, max=127752, avg=2262.01, stdev=776.42
clat percentiles (nsec):
| 1.00th=[ 1640], 5.00th=[ 1672], 10.00th=[ 1720], 20.00th=[ 1752],
| 30.00th=[ 1800], 40.00th=[ 1896], 50.00th=[ 2256], 60.00th=[ 2352],
| 70.00th=[ 2416], 80.00th=[ 2448], 90.00th=[ 2640], 95.00th=[ 3280],
| 99.00th=[ 4320], 99.50th=[ 4704], 99.90th=[ 7840], 99.95th=[10816],
| 99.99th=[18304]
lat (usec) : 2=47.06%, 4=51.10%, 10=1.78%, 20=0.06%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=10.08%, sys=88.89%, ctx=38, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1320MiB/s (1384MB/s), 1320MiB/s-1320MiB/s (1384MB/s-1384MB/s), io=512MiB (537MB), run=388-388msec
/ #
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment