ext4 JBOD 4x PCIe SSD

fio.out

atobey@zorak ~ $ sudo fio jbod-rw5050.fio
rw5050-sdi: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
rw5050-sdj: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
rw5050-sdk: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
rw5050-sdl: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.9
Starting 4 processes
rw5050-sdi: Laying out IO file(s) (1 file(s) / 25600MB)
rw5050-sdj: Laying out IO file(s) (1 file(s) / 25600MB)
rw5050-sdk: Laying out IO file(s) (1 file(s) / 25600MB)
rw5050-sdl: Laying out IO file(s) (1 file(s) / 25600MB)
Jobs: 4 (f=4): [mmmm] [100.0% done] [260.2MB/259.7MB/0KB /s] [66.6K/66.5K/0 iops] [eta 00m:00s]
rw5050-sdi: (groupid=0, jobs=4): err= 0: pid=57435: Tue Jun 17 21:57:26 2014
  Description  : ["random 50/50 sdi"]
  read : io=154783MB, bw=264163KB/s, iops=66040, runt=600001msec
    slat (usec): min=24, max=6051, avg=90.57, stdev=189.80
    clat (usec): min=0, max=784536, avg=919.92, stdev=1161.67
     lat (usec): min=66, max=784588, avg=1010.51, stdev=1172.56
    clat percentiles (usec):
     |  1.00th=[  179],  5.00th=[  262], 10.00th=[  322], 20.00th=[  398],
     | 30.00th=[  454], 40.00th=[  510], 50.00th=[  580], 60.00th=[  668],
     | 70.00th=[  836], 80.00th=[ 1128], 90.00th=[ 2704], 95.00th=[ 2928],
     | 99.00th=[ 3216], 99.50th=[ 3408], 99.90th=[ 5088], 99.95th=[ 5664],
     | 99.99th=[ 9024]
    bw (KB  /s): min=    5, max=89392, per=25.00%, avg=66047.78, stdev=7765.88
  write: io=154857MB, bw=264289KB/s, iops=66072, runt=600001msec
    slat (usec): min=25, max=6051, avg=91.76, stdev=192.06
    clat (usec): min=0, max=784307, avg=814.77, stdev=1110.86
     lat (usec): min=54, max=784356, avg=906.55, stdev=1123.42
    clat percentiles (usec):
     |  1.00th=[   73],  5.00th=[  171], 10.00th=[  245], 20.00th=[  346],
     | 30.00th=[  414], 40.00th=[  474], 50.00th=[  532], 60.00th=[  612],
     | 70.00th=[  732], 80.00th=[  972], 90.00th=[ 2480], 95.00th=[ 2800],
     | 99.00th=[ 3088], 99.50th=[ 3248], 99.90th=[ 4640], 99.95th=[ 5472],
     | 99.99th=[ 9024]
    bw (KB  /s): min=    3, max=87952, per=25.00%, avg=66080.22, stdev=7772.54
    lat (usec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11%
    lat (usec) : 100=0.97%, 250=6.04%, 500=34.14%, 750=27.03%, 1000=10.14%
    lat (msec) : 2=8.72%, 4=12.53%, 10=0.22%, 20=0.01%, 50=0.01%
    lat (msec) : 750=0.01%, 1000=0.01%
  cpu          : usr=1.82%, sys=22.32%, ctx=49333726, majf=0, minf=97
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=100.8%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=39624417/w=39643311/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=154783MB, aggrb=264162KB/s, minb=264162KB/s, maxb=264162KB/s, mint=600001msec, maxt=600001msec
  WRITE: io=154857MB, aggrb=264288KB/s, minb=264288KB/s, maxb=264288KB/s, mint=600001msec, maxt=600001msec

Disk stats (read/write):
  sdi: ios=9560080/9562055, merge=11/124, ticks=7104147/6810767, in_queue=13910830, util=98.03%
  sdj: ios=9387483/9390577, merge=20/137, ticks=6998063/6713816, in_queue=13706540, util=97.58%
  sdk: ios=9503251/9504454, merge=9/129, ticks=7085920/6786777, in_queue=13868587, util=97.89%
  sdl: ios=11488400/11501230, merge=19/152, ticks=9647390/2849493, in_queue=12491600, util=98.20%