I've had a chance to test these suggestions again on an EC2 i2.4xlarge
instance. atime=off
made no difference, zfs_prefetch_disable=1
made no difference either.
Running iotop
I am not seeing substantial read workloads; it's all writes as expected.
I can't get more than ~65K IOPS.
For lack of something else to try, here are the details of the most recent test I ran:
I set up the test with these commands: