On Tue Oct 27, 2015, history.state.gov began buckling under load, intermittently issuing 500 errors. Nginx's error log was sprinkled with the following errors:
2015/10/27 21:48:36 [crit] 2475#0: accept4() failed (24: Too many open files)
2015/10/27 21:48:36 [alert] 2475#0: *7163915 socket() failed (24: Too many open files) while connecting to upstream...
An article at http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/ provided directions that mostly worked. Below are the steps we followed. The steps that diverged from the article's directions are marked with an *.
- * Instead of using
ulimiton the nginx account, use
ps aux | grep nginxto locate nginx's process IDs. Then query each process's file handle limits using
pidis the process id retrieved from
sudomay be necessary on your system for the
catcommand here, depending on your system.)
fs.file-max = 70000to /etc/sysctl.conf