The benchmark for concurrent sequential writes to S3 with fio

seqwrite_fio.log

$ fio seq_write_1.fio; fio seq_write_2.fio; fio seq_write_4.fio; fio seq_write_8.fio;
write_bench_1: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
fio-3.21
Starting 1 process
Jobs: 1 (f=1): [W(1)][98.1%][w=3072KiB/s][w=3 IOPS][eta 00m:01s]
write_bench_1: (groupid=0, jobs=1): err= 0: pid=46386: Wed Aug 12 17:43:37 2020
  write: IOPS=1, BW=1943KiB/s (1990kB/s)(100MiB/52690msec); 0 zone resets
    clat (msec): min=278, max=3319, avg=526.77, stdev=400.93
     lat (msec): min=278, max=3319, avg=526.88, stdev=400.93
    clat percentiles (msec):
     |  1.00th=[  279],  5.00th=[  363], 10.00th=[  384], 20.00th=[  393],
     | 30.00th=[  405], 40.00th=[  422], 50.00th=[  435], 60.00th=[  456],
     | 70.00th=[  468], 80.00th=[  506], 90.00th=[  584], 95.00th=[  760],
     | 99.00th=[ 2567], 99.50th=[ 3306], 99.90th=[ 3306], 99.95th=[ 3306],
     | 99.99th=[ 3306]
   bw (  KiB/s): min= 2043, max= 4096, per=100.00%, avg=2299.37, stdev=680.47, samples=88
   iops        : min=    1, max=    4, avg= 1.40, stdev= 0.80, samples=88
  lat (msec)   : 500=77.00%, 750=17.00%, 1000=2.00%, 2000=2.00%, >=2000=2.00%
  cpu          : usr=3.36%, sys=0.33%, ctx=268, majf=0, minf=264
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,100,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1943KiB/s (1990kB/s), 1943KiB/s-1943KiB/s (1990kB/s-1990kB/s), io=100MiB (105MB), run=52690-52690msec
write_bench_1/2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_2/2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
fio-3.21
Starting 2 processes
Jobs: 1 (f=1): [_(1),W(1)][98.0%][w=1026KiB/s][w=1 IOPS][eta 00m:02s]
write_bench_1/2: (groupid=0, jobs=2): err= 0: pid=46402: Wed Aug 12 17:45:19 2020
  write: IOPS=1, BW=2035KiB/s (2083kB/s)(200MiB/100663msec); 0 zone resets
    clat (msec): min=499, max=6464, avg=1000.02, stdev=582.73
     lat (msec): min=499, max=6464, avg=1000.14, stdev=582.73
    clat percentiles (msec):
     |  1.00th=[  550],  5.00th=[  617], 10.00th=[  651], 20.00th=[  709],
     | 30.00th=[  751], 40.00th=[  802], 50.00th=[  860], 60.00th=[  936],
     | 70.00th=[ 1003], 80.00th=[ 1133], 90.00th=[ 1301], 95.00th=[ 1921],
     | 99.00th=[ 3071], 99.50th=[ 4245], 99.90th=[ 6477], 99.95th=[ 6477],
     | 99.99th=[ 6477]
   bw (  KiB/s): min= 4070, max= 4096, per=100.00%, avg=4090.33, stdev= 2.69, samples=198
   iops        : min=    2, max=    4, avg= 2.90, stdev= 0.50, samples=198
  lat (msec)   : 500=0.50%, 750=30.00%, 1000=39.00%, 2000=26.00%, >=2000=4.50%
  cpu          : usr=1.87%, sys=0.18%, ctx=1390, majf=0, minf=507
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2035KiB/s (2083kB/s), 2035KiB/s-2035KiB/s (2083kB/s-2083kB/s), io=200MiB (210MB), run=100663-100663msec
write_bench_1/4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_2/4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_3/4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_4/4: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
fio-3.21
Starting 4 processes
Jobs: 1 (f=1): [W(1),_(3)][99.1%][w=2048KiB/s][w=2 IOPS][eta 00m:02s]
write_bench_1/4: (groupid=0, jobs=4): err= 0: pid=46442: Wed Aug 12 17:48:57 2020
  write: IOPS=1, BW=1891KiB/s (1937kB/s)(400MiB/216570msec); 0 zone resets
    clat (msec): min=511, max=6149, avg=2133.07, stdev=873.96
     lat (msec): min=511, max=6149, avg=2133.19, stdev=873.96
    clat percentiles (msec):
     |  1.00th=[  609],  5.00th=[ 1083], 10.00th=[ 1250], 20.00th=[ 1435],
     | 30.00th=[ 1603], 40.00th=[ 1754], 50.00th=[ 1955], 60.00th=[ 2165],
     | 70.00th=[ 2433], 80.00th=[ 2735], 90.00th=[ 3272], 95.00th=[ 3708],
     | 99.00th=[ 5067], 99.50th=[ 5873], 99.90th=[ 6141], 99.95th=[ 6141],
     | 99.99th=[ 6141]
   bw (  KiB/s): min= 8172, max= 8937, per=100.00%, avg=8187.10, stdev=37.57, samples=395
   iops        : min=    4, max=    8, avg= 5.50, stdev= 0.48, samples=395
  lat (msec)   : 750=1.50%, 1000=2.25%, 2000=48.00%, >=2000=48.25%
  cpu          : usr=0.92%, sys=0.11%, ctx=4125, majf=0, minf=989
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1891KiB/s (1937kB/s), 1891KiB/s-1891KiB/s (1937kB/s-1937kB/s), io=400MiB (419MB), run=216570-216570msec
write_bench_1/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_2/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_3/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_4/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_5/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_6/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_7/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
write_bench_8/8: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=http, iodepth=1
fio-3.21
Starting 8 processes
Jobs: 1 (f=1): [_(3),W(1),_(4)][99.2%][w=2050KiB/s][w=2 IOPS][eta 00m:03s]
write_bench_1/8: (groupid=0, jobs=8): err= 0: pid=46525: Wed Aug 12 17:54:50 2020
  write: IOPS=2, BW=2332KiB/s (2388kB/s)(800MiB/351300msec); 0 zone resets
    clat (msec): min=816, max=5820, avg=3445.37, stdev=745.27
     lat (msec): min=816, max=5820, avg=3445.47, stdev=745.27
    clat percentiles (msec):
     |  1.00th=[ 1368],  5.00th=[ 2299], 10.00th=[ 2534], 20.00th=[ 2836],
     | 30.00th=[ 3071], 40.00th=[ 3272], 50.00th=[ 3440], 60.00th=[ 3608],
     | 70.00th=[ 3809], 80.00th=[ 4044], 90.00th=[ 4396], 95.00th=[ 4665],
     | 99.00th=[ 5269], 99.50th=[ 5470], 99.90th=[ 5805], 99.95th=[ 5805],
     | 99.99th=[ 5805]
   bw (  KiB/s): min=14990, max=16384, per=100.00%, avg=16346.52, stdev=47.79, samples=792
   iops        : min=    8, max=   16, avg=13.97, stdev= 0.43, samples=792
  lat (msec)   : 1000=0.25%, 2000=2.12%, >=2000=97.62%
  cpu          : usr=0.43%, sys=0.06%, ctx=12976, majf=0, minf=1993
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2332KiB/s (2388kB/s), 2332KiB/s-2332KiB/s (2388kB/s-2388kB/s), io=800MiB (839MB), run=351300-351300msec