Postgres Filesystem Benchmark of a Corsair MP600 500GB NVMe SSD mounted in a PCIe 2.0 x1 slot (while it would support PCIe 4.0 x4)

pg_test_fsync

$ uname -a
Linux owl 5.5.8-arch1-1 #1 SMP PREEMPT Fri, 06 Mar 2020 00:57:33 +0000 x86_64 GNU/Linux
$ cat /etc/os-release 
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
$ pg_config --version
PostgreSQL 12.2

$ pg_test_fsync
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                      2286.186 ops/sec     437 usecs/op
        fdatasync                          2216.011 ops/sec     451 usecs/op
        fsync                              1904.720 ops/sec     525 usecs/op
        fsync_writethrough                              n/a
        open_sync                          1900.989 ops/sec     526 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                      1137.062 ops/sec     879 usecs/op
        fdatasync                          2160.299 ops/sec     463 usecs/op
        fsync                              1896.817 ops/sec     527 usecs/op
        fsync_writethrough                              n/a
        open_sync                           958.446 ops/sec    1043 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
         1 * 16kB open_sync write          1917.688 ops/sec     521 usecs/op
         2 *  8kB open_sync writes          957.965 ops/sec    1044 usecs/op
         4 *  4kB open_sync writes          487.403 ops/sec    2052 usecs/op
         8 *  2kB open_sync writes          194.621 ops/sec    5138 usecs/op
        16 *  1kB open_sync writes           97.857 ops/sec   10219 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
        write, fsync, close                1884.696 ops/sec     531 usecs/op
        write, close, fsync                1891.182 ops/sec     529 usecs/op

Non-sync'ed 8kB writes:
        write                            576714.823 ops/sec       2 usecs/op