valyala/mmap.md Secret

## mmap.md

      
    Raw
  

              mmap.md
            
          
    mmap in Go considered harmful

Do you use syscall.Mmap
in Go?
There are high chances the answer is yes even if your are unware of it.
Your app's direct or indirect dependencies may use syscall.Mmap
because of a widespread beleif - "mmap is faster than plain old file I/O".
Let's try to understand whether this is is true.
What is mmap?

mmap is a system call
for mapping file contents into memory. After the mapping, you can read and/or
write file contents by just accessing memory region returned by the syscall.
Convenient, isn't it? There is no need in heavy system calls
for reading
and/or writing
the file contents. Absolute victory? No!
How does mmap work?

What happens when a program accesses memory address inside the region
returned by mmap? There are two cases:

The given memory address points to a hot data already present
in memory. Such memory is known as page cache.
In this case the access may be indeed faster than the access via read / write syscall.
The given memory address points to a cold data missing in a page cache.
In this case the Operating Sytem (OS) intercepts
the memory access via Major page fault,
loads the requested data from the mmap'ed file into page cache and then returns
the control to the program. All this magic is invisible to the program - it just accesses
data at the specified memory location as usual. But it has very high price - cold
data access requires 100000x more time than the hot data access. Why?
See Latency Numbers Every Programmer Should Know.

I hear your voices - "read / write syscalls has almost the same price for cold
data access as major page faults for mmaped file - the implicit memory interception
is just substituted by an explicit system call".
Yes. But let's take a closer look into Go runtime.
What's wrong with mmap in Go?

Go runtime runs goroutines on OS threads.
GOMAXPROCS goroutines can be executed simultaneously by OS threads. Other ready goroutines wait
for their turn until the currently running goroutines block, yield or stuck in cgo call / syscall.
Goroutines may block on I/O, channel, mutex. Goroutines may yield on function call, memory allocation or explicit
runtime.Gosched call. Goroutines don't block on a major page fault!
Again - goroutines don't block and don't yield on a major page fault, since it is invisible to Go runtime.
What happens when a goroutine accesses cold data in mmaped file? It stucks for a looooong time.
During this time it occupies an OS thread from GOMAXPROCS threads, so other ready goroutines have reduced
number of threads to be executed on. This leads to CPU underutilization.
What happens if GOMAXPROCS goroutines concurrently access cold data regions in mmaped file?
Complete stall of the whole program until the OS resolves major page faults caused by these goroutines!
How to detect stalls caused by major page faults in Go programs?

Monitor request latencies and CPU usage:

latencies for ALL the requests usually increase during stalls;
user CPU share drops, since the program performs less work during stalls;
system and iowait CPU shares increase because the OS handles major page faults.

How to deal with these stalls?


Increase GOMAXPROCS to N x NumCPU. This reduces chances of CPU underutilization
during major page faults at the cost of higher CPU usage, since now each CPU core deals with multiple OS threads.
Access mmaped data only via cgo calls. Go launches an additional OS thread for each goroutine stuck inside cgo call.
This prevents from CPU underutilization at the cost of higher CPU usage, since cgo calls are expensive.
Do not use mmap in Go programs. This solution has no drawbacks.

There are many Go programs with mmap inside and nobody complains.

These programs may work without issues until the accessed data from mmaped file
fits page cache. Page cache size is limited by RAM size. So these programs should
experience stalls when mmaped files exceed RAM size. Program stalls
may be left unnoticed on low loads and on faster storages (SSDs instead of HDDs or network storages).
The program may experience stalls if mmaped file is smaller than the RAM size in the following cases:

On the first access to mmaped data if it isn't present in the page cache yet.
Such stalls usually occur during program warmup. Note that stalls induced by major page fault
increase latencies for ALL the requests, including requests, which don't touch mmaped data.
On the first access to mmaped data after its' eviction from the page cache. The eviction may be caused
by a third-party app running on the same OS. For instance, innocent grep over a big log file quickly
evicts useful data from page cache.

Conclusion

Avoid mmap inside Go programs, since it may cause stalls.
Read how we were creating the best remote storage for Prometheus.
It is written in Go and it doesn't use mmap :)