Skip to content

Instantly share code, notes, and snippets.

@v0lkan
Last active April 2, 2024 18:25
Show Gist options
  • Save v0lkan/90fcb83c86918732b894 to your computer and use it in GitHub Desktop.
Save v0lkan/90fcb83c86918732b894 to your computer and use it in GitHub Desktop.
Configuring NGINX for Maximum Throughput Under High Concurrency
user web;
# One worker process per CPU core.
worker_processes 8;
# Also set
# /etc/security/limits.conf
# web soft nofile 65535
# web hard nofile 65535
# /etc/default/nginx
# ULIMIT="-n 65535"
worker_rlimit_nofile 65535;
pid /run/nginx.pid;
events {
#
# Determines how many clients will be served by each worker process.
# (Max clients = worker_connections * worker_processes)
# Should be equal to `ulimit -n / worker_processes`
#
worker_connections 65535;
#
# Let each process accept multiple connections.
# Accept as many connections as possible, after nginx gets notification
# about a new connection.
# May flood worker_connections, if that option is set too low.
#
multi_accept on;
#
# Preferred connection method for newer linux versions.
# Essential for linux, optmized to serve many clients with each thread.
#
use epoll;
}
http {
##
# Basic Settings
##
#
# Override some buffer limitations, will prevent DDOS too.
#
client_body_buffer_size 10K;
client_header_buffer_size 1k;
client_max_body_size 8m;
large_client_header_buffers 2 1k;
#
# Timeouts
# The client_body_timeout and client_header_timeout directives are
# responsible for the time a server will wait for a client body or
# client header to be sent after request. If neither a body or header
# is sent, the server will issue a 408 error or Request time out.
#
# The keepalive_timeout assigns the timeout for keep-alive connections
# with the client. Simply put, Nginx will close connections with the
# client after this period of time.
#
# Finally, the send_timeout is a timeout for transmitting a response
# to the client. If the client does not receive anything within this
# time, then the connection will be closed.
#
#
# send the client a "request timed out" if the body is not loaded
# by this time. Default 60.
#
client_body_timeout 32;
client_header_timeout 32;
#
# Every 60 seconds server broadcasts Sync packets, so 90 is
# a conservative upper bound.
#
keepalive_timeout 90; # default 65
send_timeout 120; # default 60
#
# Allow the server to close the connection after a client stops
# responding.
# Frees up socket-associated memory.
#
reset_timedout_connection on;
#
# Open file descriptors.
# Caches information about open FDs, freqently accessed files.
#
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
#
# Sendfile copies data between one FD and other from within the kernel.
# More efficient than read() + write(), since the requires transferring
# data to and from the user space.
#
sendfile on;
# Tcp_nopush causes nginx to attempt to send its HTTP response head in one
# packet, instead of using partial frames. This is useful for prepending
# headers before calling sendfile, or for throughput optimization.
tcp_nopush on;
#
# don't buffer data-sends (disable Nagle algorithm). Good for sending
# frequent small bursts of data in real time.
#
tcp_nodelay on;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
#
# Use analytics to track stuff instead of using precious file IO resources.
# Disabling logging speeds up IO.
#
access_log off;
error_log /root/PROJECTS/logs/error.log crit;
##
# Gzip Settings
##
gzip on;
gzip_disable "MSIE [1-6]\.";
# Only allow proxy request with these headers to be gzipped.
gzip_proxied expired no-cache no-store private auth;
# Default is 6 (1<n<9), but 2 -- even 1 -- is enough. The higher it is, the
# more CPU cycles will be wasted.
gzip_comp_level 9;
gzip_min_length 500; # Default 20
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
}
@v0lkan
Copy link
Author

v0lkan commented Sep 3, 2022

@Altroo some cloud vendors might not allow you modify file handler limits; that might be the issue.

@jjxtra
Copy link

jjxtra commented Feb 11, 2023

Nobody talks about this, but you MUST increase the file max descriptors per process, as for very historical reasons, Linux limits to 1024 even in modern distros...

sudo nano /etc/security/limits.conf
# add:
* - nofile 65536
root - nofile 65536

sudo nano /etc/systemd/system.conf
# add:
DefaultLimitNOFILE=65536

sudo nano /etc/sysctl.conf
# add:
fs.file-max = 65536

nano /etc/default/nginx
# add:
ULIMIT="-n 65536"

Then reboot.

@fapdash
Copy link

fapdash commented Apr 3, 2023

@v0lkan @jjxtra Why are you setting the soft limit on open files equal to the hard limit on open files? Will the worker processes not up their limit by themselves until they hit the hard limit?

Edit:

https://www.freedesktop.org/software/systemd/man/systemd.exec.html says regarding LimitNOFILE:

Nowadays, the hard limit defaults to 524288, a very high value compared to historical defaults. Typically applications should increase their soft limit to the hard limit on their own, if they are OK with working with file descriptors above 1023, i.e. do not use select(2).

But I'm still not sure if that means that nginx should do that on it's own or if we have to set that explicitly via worker_rlimit_nofile. From my tests it seems that worker_rlimit_nofile sets both the soft limit and the hard limit to that value.


For what it's worth on a Ubuntu Server 22.04 machine the values I get are:

$ cat /proc/sys/fs/file-max
9223372036854775807
$ ulimit -Hn
1048576

@fapdash
Copy link

fapdash commented Apr 3, 2023

And I've read online a couple of times now that worker_rlimit_nofile should be double the worker_connections if you are using nginx as an reverse proxy because a revers proxy connection needs 2 file handles, is not not true?

@jjxtra
Copy link

jjxtra commented Apr 3, 2023

I ended up using 1000000 for most of these values and it was fine. For nginx you can then set the worker limit to 2000000.

@v0lkan
Copy link
Author

v0lkan commented Apr 18, 2023

@v0lkan @jjxtra Why are you setting the soft limit on open files equal to the hard limit on open files?

That is an arbitrary choice.

You can make hard limit larger than the soft limit.

As long as they are large enough (and you have enough resources) things should work fine.

--

@ everyone else

I am typically terribly busy, and only have fraction of a time to look at and maintain this gist.

And it has been a long time since I last updated it; please cross-check this one with the latest NGINX documentation.

I don't think much have changed since then, since these are pretty low-level config; but still, the only time I will likely maintain this document will be when I need to scale NGINX to its limits again :) — which may be five or ten years from now; or maybe never :).

@v0lkan
Copy link
Author

v0lkan commented Apr 18, 2023

Nobody talks about this, but you MUST increase the file max descriptors per process, as for very historical reasons, Linux limits to 1024 even in modern distros...

Yes, I forgot to mention that. I assume that the reader would know about that.

In addition certain cloud vendors do not allow fiddling with the limits; so always verify that you have the limits as expected after you set them.

@hummingbird-12
Copy link

hummingbird-12 commented Jul 21, 2023

Hi @v0lkan,

Unlike for the access_log directive, error_log off; does not disable the error logs.
error_log off; would create a file named "off" for logging the errors within the default NGINX config files directory.

This is also mentioned in the NGINX blog post, in which it recommends specifying error_log /dev/null emerg; in cases where "disabling" error logging is preferred :)

Thanks! :)

@RaneemAlRushud
Copy link

Good day @arnav13081994,

Yes, you've.

"client_body_buffer_size 10K;"
This directive sets the buffer size for storing the client request body. In this case, it is set to 10 kilobytes (10k). It determines the maximum amount of data that Nginx can temporarily store from the client while the request is being processed.

Since the client_max_body_size directive is set to 16 megabytes (miga), you should choose a client_body_buffer_size value that accommodates this maximum size.

Considering that Nginx uses memory for other purposes as well, you can allocate a buffer size slightly larger than the expected maximum request body size.

@v0lkan
Copy link
Author

v0lkan commented Dec 20, 2023

Folks;

I haven’t been updating this for a while, and this file should be considered a starting point, not the end-all-be-all.

Every production setup and server is different; you’ll need to benchmark your performance, tweak things on the go, and read the official docs.

Nowadays, I’m dealing with pure NGINX less and cloud load balancers more;
however, if I see anything worthwhile to add here or any change that can improve in a system-agnostic way, I’ll update the file.

One other remark: This file's particular use was for a very high-throughput system with relatively smaller payloads. If you have larger payloads, your memory utilization will increase, and having too high of a file limit can indeed crash the process due to memory pressure.

So, yep, lots of moving parts, yet I tried to document the file as best as I can.

Cheers.

@Pvpsmash
Copy link

Hi, checking up, i run a pterodactyl.io hosting company (MCST.IO), and ive ran into some issues regarding gzip for plugin module's for minecraft, im unsure if the issue is gzip or php-fpm, but in general should gzip just be disabled from nginx.conf ? unsure as default conf does not match my other node server configs at default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment