Skip to content

Instantly share code, notes, and snippets.

@mrnugget
Created August 2, 2021 08:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mrnugget/b6bff62b3265761e640e9124947fe293 to your computer and use it in GitHub Desktop.
Save mrnugget/b6bff62b3265761e640e9124947fe293 to your computer and use it in GitHub Desktop.
Notes from debugging difference in Postgres performance between 16in MBP and Linux machine with roughly same specs.

pg_test_fsync outputs

On macOS on the 16" MBP:

pg_test_fsync
5 seconds per test
Direct I/O is not supported on this platform.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                     27492.613 ops/sec      36 usecs/op
        fdatasync                         26373.551 ops/sec      38 usecs/op
        fsync                             26449.921 ops/sec      38 usecs/op
        fsync_writethrough                   41.503 ops/sec   24095 usecs/op
        open_sync                         27028.415 ops/sec      37 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                     12833.442 ops/sec      78 usecs/op
        fdatasync                         22520.738 ops/sec      44 usecs/op
        fsync                             22718.435 ops/sec      44 usecs/op
        fsync_writethrough                   39.467 ops/sec   25338 usecs/op
        open_sync                         12263.507 ops/sec      82 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
         1 * 16kB open_sync write         24691.753 ops/sec      40 usecs/op
         2 *  8kB open_sync writes        12787.775 ops/sec      78 usecs/op
         4 *  4kB open_sync writes         6507.818 ops/sec     154 usecs/op
         8 *  2kB open_sync writes         3205.598 ops/sec     312 usecs/op
        16 *  1kB open_sync writes         1617.759 ops/sec     618 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
        write, fsync, close                9988.357 ops/sec     100 usecs/op
        write, close, fsync               18553.995 ops/sec      54 usecs/op

Non-sync'ed 8kB writes:
        write                             26806.255 ops/sec      37 usecs/op

On Linux:

/usr/lib/postgresql/12/bin/pg_test_fsync                                                                                                                                                                -127-
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                       568.044 ops/sec    1760 usecs/op
        fdatasync                          1307.635 ops/sec     765 usecs/op
        fsync                               710.051 ops/sec    1408 usecs/op
        fsync_writethrough                              n/a
        open_sync                           718.734 ops/sec    1391 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                       283.945 ops/sec    3522 usecs/op
        fdatasync                          1108.933 ops/sec     902 usecs/op
        fsync                               640.977 ops/sec    1560 usecs/op
        fsync_writethrough                              n/a
        open_sync                           357.980 ops/sec    2793 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
         1 * 16kB open_sync write           645.170 ops/sec    1550 usecs/op
         2 *  8kB open_sync writes          356.379 ops/sec    2806 usecs/op
         4 *  4kB open_sync writes          188.997 ops/sec    5291 usecs/op
         8 *  2kB open_sync writes           88.864 ops/sec   11253 usecs/op
        16 *  1kB open_sync writes           44.061 ops/sec   22696 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
        write, fsync, close                 714.729 ops/sec    1399 usecs/op
        write, close, fsync                 709.597 ops/sec    1409 usecs/op

Non-sync'ed 8kB writes:
        write                            814553.097 ops/sec       1 usecs/op

https://askubuntu.com/questions/87035/how-to-check-hard-disk-performance


Testing with fio

Sequential READ speed with big blocks:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting

Sequential WRITE speed with big blocks:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting

Random 4K read QD1:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randread --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

Mixed random 4K read and write QD1 with sync:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randrw --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

Linux with Samsung 970 EVO Plus

Sequential READ:

READ: bw=3409MiB/s (3574MB/s), 3409MiB/s-3409MiB/s (3574MB/s-3574MB/s), io=10.0GiB (10.7GB), run=3004-3004msec

Sequential WRITE:

WRITE: bw=3126MiB/s (3278MB/s), 3126MiB/s-3126MiB/s (3278MB/s-3278MB/s), io=10.0GiB (10.7GB), run=3276-3276msec

Random 4k read QD1:

READ: bw=54.8MiB/s (57.4MB/s), 54.8MiB/s-54.8MiB/s (57.4MB/s-57.4MB/s), io=3287MiB (3447MB), run=60001-60001msec

Mixed random 4k read and write QD1 with fsync:

READ: bw=1074KiB/s (1100kB/s), 1074KiB/s-1074KiB/s (1100kB/s-1100kB/s), io=62.9MiB (65.0MB), run=60002-60002msec
WRITE: bw=1071KiB/s (1096kB/s), 1071KiB/s-1071KiB/s (1096kB/s-1096kB/s), io=62.7MiB (65.8MB), run=60002-60002msec

macOS, MacBook Pro 16", 1TB SSD

(Requires sudo and --ioengine=posixaio in the fio commands above)

Sequential READ:

READ: bw=3232MiB/s (3389MB/s), 3232MiB/s-3232MiB/s (3389MB/s-3389MB/s), io=10.0GiB (10.7GB), run=3168-3168msec

Sequential WRITE:

WRITE: bw=3174MiB/s (3328MB/s), 3174MiB/s-3174MiB/s (3328MB/s-3328MB/s), io=10.0GiB (10.7GB), run=3226-3226msec

Random 4k read QD1:

READ: bw=29.2MiB/s (30.6MB/s), 29.2MiB/s-29.2MiB/s (30.6MB/s-30.6MB/s), io=1753MiB (1838MB), run=60001-60001msec

Mixed random 4k read and write QD1 with fsync:

Run status group 0 (all jobs):
   READ: bw=14.2MiB/s (14.9MB/s), 14.2MiB/s-14.2MiB/s (14.9MB/s-14.9MB/s), io=852MiB (894MB), run=60001-60001msec
  WRITE: bw=14.1MiB/s (14.8MB/s), 14.1MiB/s-14.1MiB/s (14.8MB/s-14.8MB/s), io=848MiB (889MB), run=60001-60001msec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment