gistfile1.txt

Measured on Intel NUC with Samsung970. From "mount -v":
  /dev/nvme0n1 on /data type xfs (rw,noatime,nodiratime,attr2,discard,inode64,logbufs=8,logbsize=32k,noquota)

The lifecycle of a read request goes from state Q (queued) to C (completed) and the number in parentheses
at the end of the line for state C is the response time in microseconds. R means read, RA means read with possible readahead.
The response times are so much better for raw.

Timestamps have the format seconds.nanoseconds and start at 0 seconds. Here I have a few examples from ~18 seconds.
  
--- buffered IO with XFS

259,0    1   164143    18.000053280 305758  Q  RA 6033112 + 8 [fio]
259,0    1   164144    18.000053961 305758  G  RA 6033112 + 8 [fio]
259,0    1   164145    18.000054203 305758  P   N [fio]
259,0    1   164146    18.000054444 305758  U   N [fio] 1
259,0    1   164147    18.000054815 305758  D  RA 6033112 + 8 (18000054815) [fio]
259,0    1   164148    18.000172062     0  C  RA 6033112 + 8 (  117247) [0]

259,0    1   164149    18.000203227 305758  Q  RA 74456208 + 8 [fio]
259,0    1   164150    18.000204835 305758  G  RA 74456208 + 8 [fio]
259,0    1   164151    18.000205320 305758  P   N [fio]
259,0    1   164152    18.000205769 305758  U   N [fio] 1
259,0    1   164153    18.000206962 305758  D  RA 74456208 + 8 (18000206962) [fio]
259,0    1   164154    18.000320740     0  C  RA 74456208 + 8 (  113778) [0]

259,0    1   164155    18.000351865 305758  Q  RA 132419352 + 8 [fio]
259,0    1   164156    18.000353541 305758  G  RA 132419352 + 8 [fio]
259,0    1   164157    18.000354012 305758  P   N [fio]
259,0    1   164158    18.000354469 305758  U   N [fio] 1
259,0    1   164159    18.000355651 305758  D  RA 132419352 + 8 (18000355651) [fio]
259,0    1   164160    18.000423357     0  C  RA 132419352 + 8 (   67706) [0]

--- O_DIRECT with XFS

259,0    3   545779    18.000021085 306020  Q   R 484157464 + 8 [fio]
259,0    3   545780    18.000021567 306020  G   R 484157464 + 8 [fio]
259,0    3   545781    18.000021753 306020  P   N [fio]
259,0    3   545782    18.000021972 306020  U   N [fio] 1
259,0    3   545783    18.000022344 306020  D   R 484157464 + 8 (18000022344) [fio]
259,0    3   545784    18.000090702     0  C   R 484157464 + 8 (   68358) [0]

259,0    3   545785    18.000098347 306020  Q   R 35708168 + 8 [fio]
259,0    3   545786    18.000098785 306020  G   R 35708168 + 8 [fio]
259,0    3   545787    18.000098971 306020  P   N [fio]
259,0    3   545788    18.000099184 306020  U   N [fio] 1
259,0    3   545789    18.000099475 306020  D   R 35708168 + 8 (18000099475) [fio]
259,0    3   545790    18.000150523     0  C   R 35708168 + 8 (   51048) [0]

--- no filesystem

259,0    1  1411797    18.000002688 305533  Q   R 340759624 + 8 [fio]
259,0    1  1411798    18.000003044 305533  G   R 340759624 + 8 [fio]
259,0    1  1411799    18.000003302 305533  D   R 340759624 + 8 (18000003302) [fio]
259,0    1  1411800    18.000018865     0  C   R 340759624 + 8 (   15563) [0]

259,0    1  1411801    18.000023653 305533  Q   R 517033448 + 8 [fio]
259,0    1  1411802    18.000024014 305533  G   R 517033448 + 8 [fio]
259,0    1  1411803    18.000024259 305533  D   R 517033448 + 8 (18000024259) [fio]
259,0    1  1411804    18.000040705     0  C   R 517033448 + 8 (   16446) [0]