Skip to content

Instantly share code, notes, and snippets.

@i3v

i3v/cdm_fio.sh

Last active Sep 8, 2020
Embed
What would you like to do?
Reproducing CrystalDiskMark tests with fio - fixes for https://unix.stackexchange.com/revisions/480191/9
#!/bin/bash
# This script is based on https://unix.stackexchange.com/revisions/480191/9 .
# The following changes proved to be necessary to make it work on CentOS 7:
# * removed disk info (model, size) - not very useful, might not work in many cases.
# * using "bw" instead of "bw_bytes" to support fio version 3.1 (those availible through yum @base)
# * escaping exclamation mark in sed command
# * the ".fiomark.txt" is not auto-removed
LOOPS=5 #How many times to run each test
SIZE=1024 #Size of each test, multiples of 32 recommended for Q32 tests to give the most accurate results.
WRITEZERO=0 #Set whether to write zeroes or randoms to testfile (random is the default for both fio and crystaldiskmark); dd benchmarks typically only write zeroes which is why there can be a speed difference.
QSIZE=$(($SIZE / 32)) #Size of Q32Seq tests
SIZE+=m
QSIZE+=m
if [ -z $1 ]; then
TARGET=$HOME
echo "Defaulting to $TARGET for testing"
else
TARGET="$1"
echo "Testing in $TARGET"
fi
echo "Configuration: Size:$SIZE Loops:$LOOPS Write Only Zeroes:$WRITEZERO
Running Benchmark, please wait...
"
fio --loops=$LOOPS --size=$SIZE --filename="$TARGET/.fiomark.tmp" --stonewall --ioengine=libaio --direct=1 --zero_buffers=$WRITEZERO --output-format=json \
--name=Bufread --loops=1 --bs=$SIZE --iodepth=1 --numjobs=1 --rw=readwrite \
--name=Seqread --bs=$SIZE --iodepth=1 --numjobs=1 --rw=read \
--name=Seqwrite --bs=$SIZE --iodepth=1 --numjobs=1 --rw=write \
--name=512kread --bs=512k --iodepth=1 --numjobs=1 --rw=read \
--name=512kwrite --bs=512k --iodepth=1 --numjobs=1 --rw=write \
--name=SeqQ32T1read --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=read \
--name=SeqQ32T1write --bs=$QSIZE --iodepth=32 --numjobs=1 --rw=write \
--name=4kread --bs=4k --iodepth=1 --numjobs=1 --rw=randread \
--name=4kwrite --bs=4k --iodepth=1 --numjobs=1 --rw=randwrite \
--name=4kQ32T1read --bs=4k --iodepth=32 --numjobs=1 --rw=randread \
--name=4kQ32T1write --bs=4k --iodepth=32 --numjobs=1 --rw=randwrite \
--name=4kQ8T8read --bs=4k --iodepth=8 --numjobs=8 --rw=randread \
--name=4kQ8T8write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite > "$TARGET/.fiomark.txt"
SEQR="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "Seqread"' | grep bw | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "Seqread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
SEQW="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "Seqwrite"' | grep bw | grep -v '_' | sed 2\!d | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "Seqwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
F12KR="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "512kread"' | grep bw | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "512kread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
F12KW="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "512kwrite"' | grep bw | grep -v '_' | sed 2\!d | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "512kwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
SEQ32R="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "SeqQ32T1read"' | grep bw | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "SeqQ32T1read"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
SEQ32W="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "SeqQ32T1write"' | grep bw | grep -v '_' | sed 2\!d | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "SeqQ32T1write"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
FKR="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kread"' | grep bw | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kread"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
FKW="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kwrite"' | grep bw | grep -v '_' | sed 2\!d | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kwrite"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
FK32R="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kQ32T1read"' | grep bw | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kQ32T1read"' | grep -m1 iops | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
FK32W="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kQ32T1write"' | grep bw | grep -v '_' | sed 2\!d | cut -d: -f2 | sed s:,::g)/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kQ32T1write"' | grep iops | sed '7!d' | cut -d: -f2 | cut -d. -f1 | sed 's: ::g')"
FK8R="$(($(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kQ8T8read"' | grep bw | sed 's/ "bw" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }')/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A15 '"name" : "4kQ8T8read"' | grep iops | sed 's/ "iops" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }' | cut -d. -f1)"
FK8W="$(($(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kQ8T8write"' | grep bw | sed 's/ "bw" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }')/1024))MB/s IOPS=$(cat "$TARGET/.fiomark.txt" | grep -A80 '"name" : "4kQ8T8write"' | grep '"iops" '| sed 's/ "iops" : //g' | sed 's:,::g' | awk '{ SUM += $1} END { print SUM }' | cut -d. -f1)"
echo -e "
Results:
\033[0;33m
Sequential Read: $SEQR
Sequential Write: $SEQW
\033[0;32m
512KB Read: $F12KR
512KB Write: $F12KW
\033[1;36m
Sequential Q32T1 Read: $SEQ32R
Sequential Q32T1 Write: $SEQ32W
\033[0;36m
4KB Read: $FKR
4KB Write: $FKW
\033[1;33m
4KB Q32T1 Read: $FK32R
4KB Q32T1 Write: $FK32W
\033[1;35m
4KB Q8T8 Read: $FK8R
4KB Q8T8 Write: $FK8W
"
# rm "$TARGET/.fiomark.txt"
rm "$TARGET/.fiomark.tmp"
@Lykos153

This comment has been minimized.

Copy link

@Lykos153 Lykos153 commented Dec 31, 2018

Added handling for whitespaces in file names for gnome auto mounts (eg. "/run/media/user/Samsung USB")

https://gist.github.com/Lykos153/f63f01c64b507f1ad984c6c0e0a59ce3

@i3v

This comment has been minimized.

Copy link
Owner Author

@i3v i3v commented Dec 31, 2018

Yup, good idea, merged it here.
Thanks!

@qqgg231

This comment has been minimized.

Copy link

@qqgg231 qqgg231 commented Jan 8, 2019

sh cdm_fio.sh


cdm_fio.sh: 15: cdm_fio.sh: SIZE+=m: not found
cdm_fio.sh: 16: cdm_fio.sh: QSIZE+=m: not found
Defaulting to /root for testing
Configuration: Size:2048 Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...

fio: size too small, must not be less than minimum block size: 2048 < 524288
cdm_fio.sh: 47: cdm_fio.sh: arithmetic expression: expecting EOF: " 5120000
 5000/1024"

any idea what can i do?

Ubuntu 18.10 x64

@mobalt

This comment has been minimized.

Copy link

@mobalt mobalt commented Mar 16, 2019

sh cdm_fio.sh


cdm_fio.sh: 15: cdm_fio.sh: SIZE+=m: not found
cdm_fio.sh: 16: cdm_fio.sh: QSIZE+=m: not found
Defaulting to /root for testing
Configuration: Size:2048 Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...

fio: size too small, must not be less than minimum block size: 2048 < 524288
cdm_fio.sh: 47: cdm_fio.sh: arithmetic expression: expecting EOF: " 5120000
 5000/1024"

any idea what can i do?

Ubuntu 18.10 x64

@qqgg231 The script is using bash-specific features. Running with sh or dash won't work.

@zacisco

This comment has been minimized.

Copy link

@zacisco zacisco commented Jul 23, 2019

add

#echo -e "\033[22;37m"
echo -e "\033[0m"

before remove tmp file for reset term color

@BobbyWibowo

This comment has been minimized.

Copy link

@BobbyWibowo BobbyWibowo commented Sep 28, 2019

Any idea why I'm getting these?

./cdm_fio.sh: line 47: 566319527
 553046/1024: syntax error in expression (error token is "553046/1024")
./cdm_fio.sh: line 49: 439445782
 429146/1024: syntax error in expression (error token is "429146/1024")
./cdm_fio.sh: line 51: 567097192
 553805/1024: syntax error in expression (error token is "553805/1024")
./cdm_fio.sh: line 53: 23818268
 23260/1024: syntax error in expression (error token is "23260/1024")
./cdm_fio.sh: line 55: 392506881
 383307/1024: syntax error in expression (error token is "383307/1024")

I'd assume those numbers were the actual results that the script were supposed to parse into human-readable format, but they were being treated as errors instead and thus failing.

My results ended up looking like this:

Results:

Sequential Read:
Sequential Write: 499MB/s IOPS=0

512KB Read:
512KB Write: 389MB/s IOPS=778

Sequential Q32T1 Read:
Sequential Q32T1 Write: 510MB/s IOPS=15

4KB Read:
4KB Write: 113MB/s IOPS=28982

4KB Q32T1 Read:
4KB Q32T1 Write: 336MB/s IOPS=86095

4KB Q8T8 Read: 338MB/s IOPS=86710
4KB Q8T8 Write: 297MB/s IOPS=76225

If it matters, I'm on KDE neon 5.16.5 (Ubuntu 18.04), with zsh 5.4.2.

@i3v

This comment has been minimized.

Copy link
Owner Author

@i3v i3v commented Oct 13, 2019

@BobbyWibowo
Not sure, sorry. Maybe it fails on something as simple as echo $((2048/1024)).

@BAGELreflex

This comment has been minimized.

Copy link

@BAGELreflex BAGELreflex commented Dec 23, 2019

@BobbyWibowo
It is failing because fio is returning two rows that match the bw pattern (bw and bw_bytes) for the --rw=read command. We need to exclude those results, using something like grep -v '_'.

@qqgg231
Your issue must be being caused by a newer version of fio than this script originally worked with, as I'm experiencing the same issue. I had to break the executions of fio into multiple commands.

Here is an updated version of this script that accounts for both of my above statements (tested in CentOS 7 minimal with vim, fio and df installed): https://gist.github.com/BAGELreflex/c04e7a25d64e989cbd9376a9134b8f6d

@Rabcor

This comment has been minimized.

Copy link

@Rabcor Rabcor commented Jan 9, 2020

I just updated the original script quite a bit, here's a link to it. (It's a bit messier though unfortunately, but the user experience is better). Also updated the original stackexchange answer. It did not deliberately include any fixes from here. I have no idea how well it will/won't work with other distros, I'm only testing it on Manjaro. I find it somewhat likely that one of the primary causes of issues is that these distros might have older versions of the applicable software (fio, df and bash) than what I am using.

Update: Isn't hindsight wonderful? I realized it's a waste not to at least try to make use of the compatibility upgrades you've been making around here, so I copied the most promising fork I could find (that would be yours, @BAGELreflex ) and added my upgrades to it, with a couple extras while I was at it (script's now compatible with dash, it's no longer fully reliant on bash). I'm fairly happy with this one. There's a bunch of tweaking I may need to do to fully complete it (and by tweaking I mean tweaking fio's settings, not the script) but otherwise this seems to be pretty much it.

I hope it's gonna be more compatible now... Check the original stack exchange answer to see the new version.

@kode54

This comment has been minimized.

Copy link

@kode54 kode54 commented Jan 10, 2020

Feature request, since I don't have the points on Stack Exchange to post a comment there:

Support paths that are mounted from /dev/mapper or /dev/ devices, as is the case in my setup, where the volumes are mounted from a LVM2 group inside a LUKS container. Thankfully, the $TARGET variable is correct, so it writes the test files to the correct location always, but it doesn't support resolving the disk model info from mapper or volume group paths.

However, the benchmark shows that performance is really great even when there's encryption on top of it.

@Rabcor

This comment has been minimized.

Copy link

@Rabcor Rabcor commented Jan 13, 2020

@kode54 I can look into supporting /dev/ devices, I'm not very familiar with encryption and logical partitions so I'm not sure what supporting paths from /dev/mapper involves (that is what /dev/mapper is for right?)

@kode54

This comment has been minimized.

Copy link

@kode54 kode54 commented Jan 14, 2020

Well, really, all it needs to do is resolve the actual host device from them for the drive info, and properly display a full name for the logs and such.

For example, it needs to resolve /dev// into as a volume group. That can be used with the volume group tools to identify which device the volume group is hosted on. In my case, this would resolve to a /dev/mapper/cryptlvm path, which means it's hosted on the LUKS device named cryptlvm.

Then /dev/mapper/ paths need to be resolved like that to use the cryptsetup tool or perhaps some /sys paths where applicable to resolve that to the partition/device that the LUKS is hosted on.

Conversely, some people have also been known to run LUKS on top of LVM, so it would be a /dev/mapper device that resolves to a /dev// path, that then resolves to a host device or partition.

It doesn't need any of this to still benchmark, the filesystem path is still valid for hosting the test file(s), and produces somewhat accurate results despite the overhead.

The only artifact as of now is that it ends up logging that the drive is called mappe. The stats are otherwise fine.

@JonMagon

This comment has been minimized.

Copy link

@JonMagon JonMagon commented Jul 13, 2020

I've tried to reproduce CrystalDiskMark's interface and its behavior with Qt and fio. If you're interested, please take a look at my repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.