|
# Command: |
|
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=nfs4.1-kernel --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 |
|
|
|
# Results: |
|
local: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 |
|
fio-2.16 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)] [98.5% done] [58508KB/19828KB/0KB /s] [14.7K/4957/0 iops] [eta 00m:01s] |
|
local: (groupid=0, jobs=1): err= 0: pid=32213: Thu Oct 24 15:51:59 2019 |
|
read : io=3070.4MB, bw=49233KB/s, iops=12308, runt= 63859msec |
|
write: io=1025.8MB, bw=16447KB/s, iops=4111, runt= 63859msec |
|
cpu : usr=4.58%, sys=27.81%, ctx=14039, majf=0, minf=9 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued : total=r=785996/w=262580/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: io=3070.4MB, aggrb=49233KB/s, minb=49233KB/s, maxb=49233KB/s, mint=63859msec, maxt=63859msec |
|
WRITE: io=1025.8MB, aggrb=16447KB/s, minb=16447KB/s, maxb=16447KB/s, mint=63859msec, maxt=63859msec |
|
|
|
Disk stats (read/write): |
|
drbd2069: ios=783644/261828, merge=0/0, ticks=2289976/767168, in_queue=3056796, util=98.48%, aggrios=785996/262616, aggrmerge=0/0, aggrticks=2296940/768536, aggrin_queue=3068200, aggrutil=98.40% |
|
dm-8: ios=785996/262616, merge=0/0, ticks=2296940/768536, in_queue=3068200, util=98.40%, aggrios=785996/294894, aggrmerge=0/3724, aggrticks=2296768/833152, aggrin_queue=3129580, aggrutil=98.44% |
|
sdb: ios=785996/294894, merge=0/3724, ticks=2296768/833152, in_queue=3129580, util=98.44% |
|
|
|
nfs3-ganesha: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s] |
|
nfs3-ganesha: (groupid=0, jobs=1): err= 0: pid=8403: Thu Oct 24 15:38:22 2019 |
|
read: IOPS=7039, BW=27.5MiB/s (28.8MB/s)(3070MiB/111644msec) |
|
bw ( KiB/s): min= 7105, max=46768, per=100.00%, avg=28836.70, stdev=6963.95, samples=218 |
|
iops : min= 1776, max=11692, avg=7209.17, stdev=1741.01, samples=218 |
|
write: IOPS=2352, BW=9410KiB/s (9636kB/s)(1026MiB/111644msec) |
|
bw ( KiB/s): min= 3056, max=16688, per=100.00%, avg=9636.51, stdev=2325.31, samples=218 |
|
iops : min= 764, max= 4172, avg=2409.10, stdev=581.33, samples=218 |
|
cpu : usr=3.64%, sys=11.65%, ctx=1195344, majf=0, minf=5 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=27.5MiB/s (28.8MB/s), 27.5MiB/s-27.5MiB/s (28.8MB/s-28.8MB/s), io=3070MiB (3219MB), run=111644-111644msec |
|
WRITE: bw=9410KiB/s (9636kB/s), 9410KiB/s-9410KiB/s (9636kB/s-9636kB/s), io=1026MiB (1076MB), run=111644-111644msec |
|
|
|
|
|
|
|
nfs3-kernel: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][100.0%][r=26.7MiB/s,w=9041KiB/s][r=6838,w=2260 IOPS][eta 00m:00s] |
|
nfs3-kernel: (groupid=0, jobs=1): err= 0: pid=9098: Thu Oct 24 15:42:51 2019 |
|
read: IOPS=6571, BW=25.7MiB/s (26.9MB/s)(3070MiB/119591msec) |
|
bw ( KiB/s): min= 6152, max=49096, per=99.95%, avg=26272.92, stdev=5969.38, samples=239 |
|
iops : min= 1538, max=12274, avg=6568.21, stdev=1492.34, samples=239 |
|
write: IOPS=2196, BW=8785KiB/s (8996kB/s)(1026MiB/119591msec) |
|
bw ( KiB/s): min= 2120, max=15976, per=99.95%, avg=8780.67, stdev=2013.95, samples=239 |
|
iops : min= 530, max= 3994, avg=2195.13, stdev=503.47, samples=239 |
|
cpu : usr=2.97%, sys=10.56%, ctx=1048046, majf=0, minf=6 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=25.7MiB/s (26.9MB/s), 25.7MiB/s-25.7MiB/s (26.9MB/s-26.9MB/s), io=3070MiB (3219MB), run=119591-119591msec |
|
WRITE: bw=8785KiB/s (8996kB/s), 8785KiB/s-8785KiB/s (8996kB/s-8996kB/s), io=1026MiB (1076MB), run=119591-119591msec |
|
|
|
|
|
|
|
nfs4.1-ganesha: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][100.0%][r=32.8MiB/s,w=10.8MiB/s][r=8399,w=2768 IOPS][eta 00m:00s] |
|
nfs4.1-ganesha: (groupid=0, jobs=1): err= 0: pid=10797: Thu Oct 24 15:54:16 2019 |
|
read: IOPS=5979, BW=23.4MiB/s (24.5MB/s)(3070MiB/131445msec) |
|
bw ( KiB/s): min= 3880, max=38472, per=99.84%, avg=23878.27, stdev=5355.92, samples=262 |
|
iops : min= 970, max= 9618, avg=5969.53, stdev=1338.99, samples=262 |
|
write: IOPS=1998, BW=7993KiB/s (8185kB/s)(1026MiB/131445msec) |
|
bw ( KiB/s): min= 1400, max=12752, per=99.83%, avg=7978.40, stdev=1781.98, samples=262 |
|
iops : min= 350, max= 3188, avg=1994.56, stdev=445.50, samples=262 |
|
cpu : usr=3.55%, sys=11.12%, ctx=976023, majf=0, minf=7 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=23.4MiB/s (24.5MB/s), 23.4MiB/s-23.4MiB/s (24.5MB/s-24.5MB/s), io=3070MiB (3219MB), run=131445-131445msec |
|
WRITE: bw=7993KiB/s (8185kB/s), 7993KiB/s-7993KiB/s (8185kB/s-8185kB/s), io=1026MiB (1076MB), run=131445-131445msec |
|
|
|
|
|
nfs4.1-kernel: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][100.0%][r=24.3MiB/s,w=8424KiB/s][r=6216,w=2106 IOPS][eta 00m:00s] |
|
nfs4.1-kernel: (groupid=0, jobs=1): err= 0: pid=11199: Thu Oct 24 15:57:19 2019 |
|
read: IOPS=5559, BW=21.7MiB/s (22.8MB/s)(3070MiB/141358msec) |
|
bw ( KiB/s): min= 344, max=54720, per=99.99%, avg=22235.90, stdev=7470.92, samples=282 |
|
iops : min= 86, max=13680, avg=5558.94, stdev=1867.73, samples=282 |
|
write: IOPS=1858, BW=7432KiB/s (7611kB/s)(1026MiB/141358msec) |
|
bw ( KiB/s): min= 104, max=18424, per=99.99%, avg=7431.33, stdev=2487.65, samples=282 |
|
iops : min= 26, max= 4606, avg=1857.79, stdev=621.91, samples=282 |
|
cpu : usr=2.83%, sys=9.91%, ctx=1624049, majf=0, minf=5 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=21.7MiB/s (22.8MB/s), 21.7MiB/s-21.7MiB/s (22.8MB/s-22.8MB/s), io=3070MiB (3219MB), run=141358-141358msec |
|
WRITE: bw=7432KiB/s (7611kB/s), 7432KiB/s-7432KiB/s (7611kB/s-7611kB/s), io=1026MiB (1076MB), run=141358-141358msec |
|
|
|
|
|
nfs4.2-ganesha: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][100.0%][r=33.6MiB/s,w=11.2MiB/s][r=8600,w=2876 IOPS][eta 00m:00s] |
|
nfs4.2-ganesha: (groupid=0, jobs=1): err= 0: pid=10072: Thu Oct 24 15:49:06 2019 |
|
read: IOPS=7135, BW=27.9MiB/s (29.2MB/s)(3070MiB/110135msec) |
|
bw ( KiB/s): min= 6616, max=42736, per=99.93%, avg=28524.19, stdev=5506.21, samples=220 |
|
iops : min= 1654, max=10684, avg=7131.03, stdev=1376.56, samples=220 |
|
write: IOPS=2384, BW=9539KiB/s (9768kB/s)(1026MiB/110135msec) |
|
bw ( KiB/s): min= 2544, max=13776, per=99.91%, avg=9530.59, stdev=1819.57, samples=220 |
|
iops : min= 636, max= 3444, avg=2382.61, stdev=454.89, samples=220 |
|
cpu : usr=4.07%, sys=12.15%, ctx=1130093, majf=0, minf=7 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=27.9MiB/s (29.2MB/s), 27.9MiB/s-27.9MiB/s (29.2MB/s-29.2MB/s), io=3070MiB (3219MB), run=110135-110135msec |
|
WRITE: bw=9539KiB/s (9768kB/s), 9539KiB/s-9539KiB/s (9768kB/s-9768kB/s), io=1026MiB (1076MB), run=110135-110135msec |
|
|
|
nfs4.2-kernel: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 |
|
fio-3.1 |
|
Starting 1 process |
|
Jobs: 1 (f=1): [m(1)][99.3%][r=22.9MiB/s,w=7544KiB/s][r=5870,w=1886 IOPS][eta 00m:01s] |
|
nfs4.2-kernel: (groupid=0, jobs=1): err= 0: pid=9623: Thu Oct 24 15:46:41 2019 |
|
read: IOPS=5352, BW=20.9MiB/s (21.9MB/s)(3070MiB/146824msec) |
|
bw ( KiB/s): min= 2448, max=39504, per=99.73%, avg=21352.34, stdev=6446.70, samples=293 |
|
iops : min= 612, max= 9876, avg=5338.05, stdev=1611.67, samples=293 |
|
write: IOPS=1788, BW=7156KiB/s (7327kB/s)(1026MiB/146824msec) |
|
bw ( KiB/s): min= 688, max=12992, per=99.73%, avg=7135.55, stdev=2169.67, samples=293 |
|
iops : min= 172, max= 3248, avg=1783.86, stdev=542.41, samples=293 |
|
cpu : usr=2.87%, sys=9.76%, ctx=1620963, majf=0, minf=6 |
|
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% |
|
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
|
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% |
|
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0 |
|
latency : target=0, window=0, percentile=100.00%, depth=64 |
|
|
|
Run status group 0 (all jobs): |
|
READ: bw=20.9MiB/s (21.9MB/s), 20.9MiB/s-20.9MiB/s (21.9MB/s-21.9MB/s), io=3070MiB (3219MB), run=146824-146824msec |
|
WRITE: bw=7156KiB/s (7327kB/s), 7156KiB/s-7156KiB/s (7327kB/s-7327kB/s), io=1026MiB (1076MB), run=146824-146824msec |
|
|
Hello,
I would suggest to provide CPU load benchmarks as well.
You will see the main source of issues with nfs-kernel-server is it consumes too many CPU.
When under Ubuntu I am close to 140%, under FreeBSD it is 20%, but NFS has a weird permission behavior with ZFS dataset shared over NFS
+++