directio.md

Random points on Direct IO

1. Files reside on disks. When they are opened, their contents are cached in the kernel space. They may also be cached in user-space by applications.

2. Reads and writes to files, by default, actually happen on their kernel-cached copy. The actual commit to the hard disk happens asynchronously, sometime later.

3. Of course, the OS kernels do ensure correctness of reads and writes in almost all the cases, while implementing optimizations to reduce the number of reads and writes from the disk sectors.

4. File caching makes file IO look fast! It may not be as fast if every read and write was done on/from the disk. For files that have a high cache hit rate, caching can be beneficial.

5. Some applications, however, might want to disable the kernel-level caching of files and have reads and writes happen directly from/on the disk. This is Direct IO.

6. Applications that have a high cache miss rate, are bound to benefit from direct IO.

7. Typically, applications that implement their own caching strategies might want to use direct IO to avoid another layer of caching at the kernel.

8. Most operating systems offer opening a file in the direct IO mode. For example, O_DIRECT is supported as a flag to be sent to the open() system call. Other operating systems might tag files with a DIRECT_IO like flag which means those files will always be read from or written to in the direct IO mode.

9. Databases are the biggest users of the direct IO mode.

Other interesting points:

1. Per [1], direct IO must be differentiated from synchronous IO. The latter involves using the O_SYNC flag (or calling fsync) which ensures that the write() call blocks until the data is committed to the disk. This has nothing to do with the kernel cache which is governed using the O_DIRECT flag.

2. [5] makes it clear that O_DIRECT does not give guarantees of O_SYNC.

3. [3] introduces the terms "read-ahead" and "write-behind". While using direct IO, we "read ahead" of the actually committing data to the disk and we write the data "behind" the return of the write() call.

4. [3] mentions that direct IO reduces the CPU utilisation of maintatining consistency between the cache and the on-disk file. So, if an application is sure to have a lot of cache misses, it is better to fallback to direct IO mode.

5. [3] Imagine that every file were cached in the kernel. We currently have a file with a high cache hit rate. The cache utilisation is optimal. Now, we try to read from a file where we are going to have a high cache miss rate. The file comes and sits in the cache, pushing out the contents of the "high cache hit rate" file. We have reduced the utilisation of the cache and will spend cycles maintaining cache consistency.

6. [3] mentions that a file will be opened in the direct IO mode only if it is exclusively requested for. If there are existing handles opened in the normal cached mode, the direct IO request is not honoured. If there are existing mmap'd handles, the direct IO request is not honoured. This seems to be the AIX behavior.

7. [2] mentions that the Oracle database has a well-defined set of files that are opened in the direct IO mode only. Similarly, files that will be mmap'd are also well-defined. This ensures there are no mmap-direct IO conflicts.

8. [6] And here's THE MAN lambasting direct IO and its users too!

References:

[1] http://www.alexonlinux.com/what-is-direct-io-anyway

[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/global_file_system/s1-manage-direct-io

[3] https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/generalprogramming/direct_io_normal.html

[4] https://news.ycombinator.com/item?id=16974033

[5] https://man7.org/linux/man-pages/man2/open.2.html

[6] https://lkml.org/lkml/2007/1/10/235