Skip to content

Instantly share code, notes, and snippets.

@denji
Last active September 11, 2021 12:20
Show Gist options
  • Save denji/8743267 to your computer and use it in GitHub Desktop.
Save denji/8743267 to your computer and use it in GitHub Desktop.
NFS Attribute Caching Performance Impact on Web Applications

A couple of days ago, I had some issues with NFS consistency, not every servers were up to date. Some servers had the good version of the file some hadn’t. However performing a ls -l seemed to fixed tempory the problem after each update (where a simple ls didn’t). Indeed issuing ls with the -l option triggers stat() where ls doesn’t, just because file attributes are called from the stats() function. I needed to investigate…

I. Story

We determined that some web virtual didn’t delivered the same content, some page weren’t updated properly. We quickly figured out that the issue was located of the NFS attribute caching, by default too long, at least in our setup. The first change we made was to enable the noac option on the client mount. However while trying to enhance the consistency of the NFS data, the performance impact was pretty high. The impact was easy to detect and reproduce. Basically everytime a page was request from the webser, the client had to request the NFS server to be sure to deliver the latest version. As I said, it’s quite easy to notice it, espacially the impact on the Time To First Bite value. You will mainly notice that the website takes a lot of time to load and all of sudden the content is quickly delivered.

Quick definition of the TTFB, stolen from Wikipedia, Time To First Byte or TTFB is a measurement that is often used as an indication of the responsiveness of a webserver or other network resources.

A bad TTFB can be related to a lot of things:

  • HTTP KeepAlive
  • Slow storage backend
  • Limited webserver connection

To mesure the Time To First Bite, I always use the following command (see example for my Website):

$ curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" -s http://sebastien-han.fr
Connect: 0,042 TTFB: 0,223 Total time: 0,224

Quite fast :)

II. Tests

Now see the performance impact by using different options/behavior to cache file attribute on a NFS mount (client side).

II.1. NOAC option

Stolen from the NFS man page.

NOAC: Use the noac mount option to achieve attribute cache coherence among multiple clients. Almost every file system operation checks file attribute information. The client keeps this information cached for a period of time to reduce network and server load. When noac is in effect, a client’s file attribute cache is disabled, so each operation that needs to check a file’s attributes is forced to go back to the server. This permits a client to see changes to a file very quickly, at the cost of many extra network operations.

$ for i in `seq 5`; do curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" -s http://one-of-the-website-I-host ; done
Connect: 0,018 TTFB: 9,265 Total time: 9,322
Connect: 0,009 TTFB: 7,150 Total time: 7,195
Connect: 0,012 TTFB: 7,172 Total time: 7,220
Connect: 0,010 TTFB: 7,082 Total time: 7,156
Connect: 0,019 TTFB: 10,663 Total time: 10,743

II.2. lookupcache=none

Stolen from the NFS man page.

If the client ignores its cache and validates every application lookup request with the server, that client can immediately detect when a new directory entry has been either created or removed by another client. You can specify this behavior using lookupcache=none. The extra NFS requests needed if the client does not cache directory entries can exact a performance penalty. Disabling lookup caching should result in less of a performance penalty than using noac, and has no effect on how the NFS client caches the attributes of files.

$ for i in `seq 5`; do curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" -s http://one-of-the-website-I-host ; done
Connect: 0,011 TTFB: 3,654 Total time: 3,696
Connect: 0,011 TTFB: 3,350 Total time: 3,392
Connect: 0,010 TTFB: 3,535 Total time: 3,581
Connect: 0,009 TTFB: 3,416 Total time: 3,460
Connect: 0,009 TTFB: 3,312 Total time: 3,356

II.3. actimeo=3

Stolen from the NFS man page.

Using actimeo sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.

  • acregmin=n, The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 3-second minimum.
  • acregmax=n, The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
  • acdirmin=n, The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 30-second minimum.
  • acdirmax=n, The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
$ for i in `seq 5`; do curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" -s http://one-of-the-website-I-host ; done
Connect: 0,010 TTFB: 2,592 Total time: 2,639
Connect: 0,010 TTFB: 1,592 Total time: 1,636
Connect: 0,010 TTFB: 1,679 Total time: 1,727
Connect: 0,010 TTFB: 1,592 Total time: 1,656
Connect: 0,010 TTFB: 1,695 Total time: 1,740

II.4. actimeo=1

$ for i in `seq 5`; do curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" -s http://one-of-the-website-I-host ; done
Connect: 0,010 TTFB: 1,726 Total time: 1,769
Connect: 0,009 TTFB: 1,739 Total time: 1,782
Connect: 0,009 TTFB: 1,704 Total time: 1,750
Connect: 0,009 TTFB: 3,136 Total time: 3,212
Connect: 0,014 TTFB: 1,730 Total time: 1,770

Use this mount option only if you have really good reason… From my experience, the noactimeo option did the job pretty well.

@nsteinmetz
Copy link

@denji is it noac or actimeo? Your conclusion seems a little bit mixed ;-)

Thanks for the test and sharing your experience !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment