Recovery from nginx "Too many open files" error on Amazon AWS Linux

post-mortem.md

On Tue Oct 27, 2015, history.state.gov began buckling under load, intermittently issuing 500 errors. Nginx's error log was sprinkled with the following errors:

2015/10/27 21:48:36 [crit] 2475#0: accept4() failed (24: Too many open files)

2015/10/27 21:48:36 [alert] 2475#0: *7163915 socket() failed (24: Too many open files) while connecting to upstream...

An article at http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/ provided directions that mostly worked. Below are the steps we followed. The steps that diverged from the article's directions are marked with an *.

1. * Instead of using su to run ulimit on the nginx account, use ps aux | grep nginx to locate nginx's process IDs. Then query each process's file handle limits using cat /proc/pid/limits (where pid is the process id retrieved from ps). (Note: sudo may be necessary on your system for the cat command here, depending on your system.)
2. Added fs.file-max = 70000 to /etc/sysctl.conf
3. Added nginx soft nofile 10000 and nginx hard nofile 30000 to /etc/security/limits.conf
4. Ran sysctl -p
5. Added worker_rlimit_nofile 30000; to /etc/nginx/nginx.conf.
6. * While the directions suggested that nginx -s reload was enough to get nginx to recognize the new settings, not all of nginx's processes received the new setting. Upon closer inspection of /proc/pid/limits (see #1 above), the first worker process still had the original S1024/H4096 limit on file handles. Even nginx -s quit didn't shut nginx down. The solution was to kill nginx with the kill pid. After restarting nginx, all of the nginx-user owned processes had the new file limit of S10000/H30000 handles.

Very handy, thanks

+1

Solved my issue thanks !

Solved my issue thanks !

Nice. One quick update, though, on my 2016.09 ALAMI (m3.medium) fs.file-max is set to 382547 out of the box, so I skipped step 2.

+1 thank so much

After making these changes, I was getting 1024 worker_connections are not enough
So I also increased worker_connections (which is limited by worker_rlimit_nofile). See: http://nginx.org/en/docs/ngx_core_module.html#worker_connections

Also saved my bottom. Much thanks.

This might be useful also https://ospi.fi/blog/centos-7-raise-nofile-limit-for-nginx.html

Awesome! Thanks 💥

Thanks a lot.

thanks man

Thank you. My problem seems to be solved.

Great ! Thanks 👍

Thanks!

Thank you, will implement. This adds load to the server. Has anyone experienced the CPU or Memory working harder?

I followed the instructions but I am getting 5 NGINX processes now. prior to the changes I only had 4 nginx processes. Why is there a 5 process labeled "Master", which was not there before?

One of the processes is labeled as "master" something that was not there before. The other 4 processes are worker processes. The workers processes have the right max Hn and Sn configuration. But the master does not.

I followed the instructions but I am getting 5 NGINX processes now. prior to the changes I only had 4 nginx processes. Why is there a 5 process labeled "Master", which was not there before?

I'm guessing you have 4 CPUs and workers are set to auto which means one per CPU. NGINX uses a master/worker model to achieve high speed throughput (async model). What you're seeing is perfectly normal.

From their website:

NGINX uses a predictable process model that is tuned to the available hardware resources:

• The master process performs the privileged operations such as reading configuration and binding to ports, and then creates a small number of child processes (the next three types).
• The cache loader process runs at startup to load the disk‑based cache into memory, and then exits. It is scheduled conservatively, so its resource demands are low.
• The cache manager process runs periodically and prunes entries from the disk caches to keep them within the configured sizes.
• The worker processes do all of the work! They handle network connections, read and write content to disk, and communicate with upstream servers.

read more at https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/

Worked for me, thank you so very much!!!!! You're awesome <3

tks!

Thanks a lot, it works 💯
Awesome...!

If you use systemd, instead of 2 you need to create an override and put LimitNOFILE=10000 in the [Service] section.

This solved my issue, in the process i noticed it only hit this limit because one of a default setting of 300s timeout. It was backing up connections which is why it hit the limit. I lowered timeout to 5s and also fixed an issue on the server it was sending to that had a issue of its own that caused the overall backup

Thanks for making this guide! Just one thing:

In point 5:
Added worker_rlimit_nofile 30000; to /etc/nginx/nginx.conf.

Please notice that:
The worker_rlimit_nofile 30000 directive goes in the main context of the config file. Outside server {...} and events {...}. Reference: http://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile

I was struggling with this because I tried to add it on the server {...} context and this didn't fail but it wasn't increasing the max open files.

awesome

Worked for us. Thanks a lot. Also, @jmg has an important insight

6. service nginx restart succeeded. No need to manually kill processes by PID.