server 1

rand-write: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [2025KB/1970KB/0KB /s] [506/492/0 iops] [eta 00m:00s]
rand-write: (groupid=0, jobs=1): err= 0: pid=8897: Sat Apr 23 08:58:03 2016
  read : io=240304KB, bw=2001.2KB/s, iops=500, runt=120033msec
    slat (usec): min=2, max=34144, avg=31.78, stdev=295.49
    clat (usec): min=257, max=103114, avg=33695.93, stdev=16835.50
     lat (usec): min=322, max=103143, avg=33728.70, stdev=16833.26
    clat percentiles (usec):
     |  1.00th=[ 1288],  5.00th=[ 5024], 10.00th=[ 9024], 20.00th=[17024],
     | 30.00th=[23936], 40.00th=[29056], 50.00th=[35072], 60.00th=[40192],
     | 70.00th=[45312], 80.00th=[50944], 90.00th=[55040], 95.00th=[58112],
     | 99.00th=[61184], 99.50th=[62208], 99.90th=[66048], 99.95th=[85504],
     | 99.99th=[94720]
    bw (KB  /s): min= 1652, max= 2472, per=100.00%, avg=2003.42, stdev=72.26
  write: io=239416KB, bw=1994.6KB/s, iops=498, runt=120033msec
    slat (usec): min=2, max=41864, avg=40.58, stdev=537.11
    clat (usec): min=181, max=128795, avg=30262.86, stdev=17055.19
     lat (usec): min=224, max=158974, avg=30304.44, stdev=17052.92
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 3920], 10.00th=[ 6880], 20.00th=[12864],
     | 30.00th=[18816], 40.00th=[24960], 50.00th=[30080], 60.00th=[35072],
     | 70.00th=[40704], 80.00th=[47872], 90.00th=[54016], 95.00th=[57088],
     | 99.00th=[60672], 99.50th=[62208], 99.90th=[78336], 99.95th=[93696],
     | 99.99th=[123392]
    bw (KB  /s): min= 1453, max= 2328, per=100.00%, avg=1996.13, stdev=77.32
    lat (usec) : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.64%
    lat (msec) : 2=1.43%, 4=2.96%, 10=8.68%, 20=14.41%, 50=53.04%
    lat (msec) : 100=18.79%, 250=0.02%
  cpu          : usr=2.03%, sys=4.39%, ctx=117099, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=60076/w=59854/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=240304KB, aggrb=2001KB/s, minb=2001KB/s, maxb=2001KB/s, mint=120033msec, maxt=120033msec
  WRITE: io=239416KB, aggrb=1994KB/s, minb=1994KB/s, maxb=1994KB/s, mint=120033msec, maxt=120033msec

Disk stats (read/write):
  vda: ios=59969/59955, merge=0/100, ticks=2013774/1818645, in_queue=3833023, util=100.00%

server 2

rand-write: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.2.8
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [2997KB/3004KB/0KB /s] [749/751/0 iops] [eta 00m:00s]
rand-write: (groupid=0, jobs=1): err= 0: pid=18783: Sat Apr 23 08:58:28 2016
  read : io=359964KB, bw=2999.2KB/s, iops=749, runt=120021msec
    slat (usec): min=2, max=23069, avg=36.96, stdev=251.93
    clat (usec): min=4, max=64439, avg=21415.90, stdev=10807.47
     lat (usec): min=224, max=64472, avg=21453.94, stdev=10805.57
    clat percentiles (usec):
     |  1.00th=[ 1240],  5.00th=[ 3376], 10.00th=[ 5984], 20.00th=[11072],
     | 30.00th=[14784], 40.00th=[18048], 50.00th=[21376], 60.00th=[24704],
     | 70.00th=[28544], 80.00th=[32640], 90.00th=[36096], 95.00th=[38144],
     | 99.00th=[40704], 99.50th=[41216], 99.90th=[44800], 99.95th=[46848],
     | 99.99th=[54016]
    bw (KB  /s): min= 2496, max= 3744, per=100.00%, avg=3001.24, stdev=99.06
  write: io=359124KB, bw=2992.2KB/s, iops=748, runt=120021msec
    slat (usec): min=3, max=52366, avg=40.77, stdev=319.60
    clat (usec): min=146, max=90693, avg=21216.92, stdev=11023.12
     lat (usec): min=169, max=90715, avg=21258.77, stdev=11020.46
    clat percentiles (usec):
     |  1.00th=[ 1160],  5.00th=[ 3184], 10.00th=[ 5344], 20.00th=[10560],
     | 30.00th=[14656], 40.00th=[18560], 50.00th=[21888], 60.00th=[24704],
     | 70.00th=[28032], 80.00th=[31872], 90.00th=[36096], 95.00th=[38144],
     | 99.00th=[40704], 99.50th=[42240], 99.90th=[61696], 99.95th=[70144],
     | 99.99th=[87552]
    bw (KB  /s): min= 2600, max= 3456, per=100.00%, avg=2993.38, stdev=93.89
    lat (usec) : 10=0.01%, 250=0.01%, 500=0.07%, 750=0.61%, 1000=0.21%
    lat (msec) : 2=1.74%, 4=4.24%, 10=11.83%, 20=27.13%, 50=54.05%
    lat (msec) : 100=0.12%
  cpu          : usr=2.53%, sys=7.59%, ctx=174162, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=89991/w=89781/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=359964KB, aggrb=2999KB/s, minb=2999KB/s, maxb=2999KB/s, mint=120021msec, maxt=120021msec
  WRITE: io=359124KB, aggrb=2992KB/s, minb=2992KB/s, maxb=2992KB/s, mint=120021msec, maxt=120021msec

Disk stats (read/write):
  vda: ios=89820/90024, merge=0/228, ticks=1912599/1899224, in_queue=3812144, util=100.00%

server 3

rand-write: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
rand-write: Laying out IO file(s) (1 file(s) / 512MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [4051KB/3952KB/0KB /s] [1012/988/0 iops] [eta 00m:00s]
rand-write: (groupid=0, jobs=1): err= 0: pid=2315: Sat Apr 23 09:10:44 2016
  read : io=480884KB, bw=4006.9KB/s, iops=1001, runt=120017msec
    slat (usec): min=0, max=16958, avg=27.46, stdev=166.99
    clat (usec): min=118, max=46142, avg=16781.19, stdev=8425.71
     lat (usec): min=247, max=46162, avg=16809.43, stdev=8424.30
    clat percentiles (usec):
     |  1.00th=[  892],  5.00th=[ 2544], 10.00th=[ 4512], 20.00th=[ 8384],
     | 30.00th=[11840], 40.00th=[14528], 50.00th=[17024], 60.00th=[20096],
     | 70.00th=[22912], 80.00th=[25472], 90.00th=[27520], 95.00th=[29056],
     | 99.00th=[30592], 99.50th=[30848], 99.90th=[31616], 99.95th=[34048],
     | 99.99th=[41728]
    bw (KB  /s): min= 3640, max= 5040, per=100.00%, avg=4008.66, stdev=121.33
  write: io=479084KB, bw=3991.9KB/s, iops=997, runt=120017msec
    slat (usec): min=0, max=21102, avg=29.03, stdev=144.38
    clat (usec): min=0, max=83967, avg=15145.29, stdev=8575.90
     lat (usec): min=0, max=83995, avg=15175.09, stdev=8577.28
    clat percentiles (usec):
     |  1.00th=[  454],  5.00th=[ 1960], 10.00th=[ 3344], 20.00th=[ 6368],
     | 30.00th=[ 9408], 40.00th=[12352], 50.00th=[15168], 60.00th=[17792],
     | 70.00th=[20352], 80.00th=[23936], 90.00th=[27008], 95.00th=[28544],
     | 99.00th=[30336], 99.50th=[30848], 99.90th=[39168], 99.95th=[57600],
     | 99.99th=[69120]
    bw (KB  /s): min= 3496, max= 4568, per=100.00%, avg=3994.79, stdev=109.81
    lat (usec) : 2=0.01%, 50=0.01%, 250=0.01%, 500=0.63%, 750=0.40%
    lat (usec) : 1000=0.92%
    lat (msec) : 2=2.84%, 4=6.06%, 10=17.68%, 20=35.74%, 50=35.69%
    lat (msec) : 100=0.03%
  cpu          : usr=3.53%, sys=7.54%, ctx=236591, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=120221/w=119771/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=480884KB, aggrb=4006KB/s, minb=4006KB/s, maxb=4006KB/s, mint=120017msec, maxt=120017msec
  WRITE: io=479084KB, aggrb=3991KB/s, minb=3991KB/s, maxb=3991KB/s, mint=120017msec, maxt=120017msec

Disk stats (read/write):
  vda: ios=120012/119931, merge=0/772, ticks=2008332/1825352, in_queue=3833904, util=100.00%