nleclerc

On Tue Oct 27, 2015, history.state.gov began buckling under load, intermittently issuing 500 errors. Nginx's error log was sprinkled with the following errors:

2015/10/27 21:48:36 [crit] 2475#0: accept4() failed (24: Too many open files)

2015/10/27 21:48:36 [alert] 2475#0: *7163915 socket() failed (24: Too many open files) while connecting to upstream...

An article at http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/ provided directions that mostly worked. Below are the steps we followed. The steps that diverged from the article's directions are marked with an *.

1. * Instead of using su to run ulimit on the nginx account, use ps aux | grep nginx to locate nginx's process IDs. Then query each process's file handle limits using cat /proc/pid/limits (where pid is the process id retrieved from ps). (Note: sudo may be necessary on your system for the cat command here, depending on your system.)
2. Added fs.file-max = 70000 to /etc/sysctl.conf

nleclerc

Note 1: The following CQ curl commands assumes a admin:admin username and password.
Note 2: For Windows/Powershell users: use two "" when doing a -F cURL command.
        Example: -F"":operation=delete""
Note 3: Quotes around name of package (or name of zip file, or jar) should be included.

Uninstall a bundle (use http://localhost:4505/system/console/bundles to access the Apache Felix web console)
        curl -u admin:admin -daction=uninstall http://localhost:4505/system/console/bundles/"name of bundle"

Install a bundle
        curl -u admin:admin -F action=install -F bundlestartlevel=20 -F