Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
NGINX tuning for best performance

Moved to git repository: https://github.com/denji/nginx-tuning

NGINX Tuning For Best Performance

For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.

First, you will need to install nginx

yum install nginx
apt install nginx

Backup your original configs and you can start reconfigure your configs. You will need to open your nginx.conf at /etc/nginx/nginx.conf with your favorite editor.

# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 100000;

# only log critical errors
error_log /var/log/nginx/error.log crit;

# provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = worker_connections * worker_processes
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;

    # optmized to serve many clients with each thread, essential for linux -- for testing environment
    use epoll;

    # accept as many connections as possible, may flood worker connections if set too low -- for testing environment
    multi_accept on;
}

# cache informations about FDs, frequently accessed files
# can boost performance, but you need to test those values
open_file_cache max=200000 inactive=20s; 
open_file_cache_valid 30s; 
open_file_cache_min_uses 2;
open_file_cache_errors on;

# to boost I/O on HDD we can disable access logs
access_log off;

# copies data between one FD and other from within the kernel
# faster then read() + write()
sendfile on;

# send headers in one peace, its better then sending them one by one 
tcp_nopush on;

# don't buffer data sent, good for small data bursts in real time
tcp_nodelay on;

# reduce the data that needs to be sent over network -- for testing environment
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/json application/xml;
gzip_disable msie6;

# allow the server to close connection on non responding client, this will free up memory
reset_timedout_connection on;

# request timed out -- default 60
client_body_timeout 10;

# if client stop responding, free up memory -- default 60
send_timeout 2;

# server will close connection after this time -- default 75
keepalive_timeout 30;

# number of requests client can make over keep-alive -- for testing environment
keepalive_requests 100000;

Now you can save config and run bottom command

nginx -s reload
/etc/init.d/nginx start|restart

If you wish to test config first you can run

nginx -t
/etc/init.d/nginx configtest

Just For Security Reason

server_tokens off;

Nginx Simple DDoS Defense

This is far away from secure DDoS defense but can slow down some small DDoS. Those configs are also in test environment and you should do your values.

# limit the number of connections per single IP
limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;

# limit the number of requests for a given session
limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s;

# zone which we want to limit by upper values, we want limit whole server
server {
    limit_conn conn_limit_per_ip 10;
    limit_req zone=req_limit_per_ip burst=10 nodelay;
}

# if the request body size is more than the buffer size, then the entire (or partial)
# request body is written into a temporary file
client_body_buffer_size  128k;

# headerbuffer size for the request header from client -- for testing environment
client_header_buffer_size 3m;

# maximum number and size of buffers for large headers to read from client request
large_client_header_buffers 4 256k;

# read timeout for the request body from client -- for testing environment
client_body_timeout   3m;

# how long to wait for the client to send a request header -- for testing environment
client_header_timeout 3m;

Now you can do again test config

nginx -t # /etc/init.d/nginx configtest

And then reload or restart your nginx

nginx -s reload
/etc/init.d/nginx reload|restart

You can test this configuration with tsung and when you are satisfied with result you can hit Ctrl+C because it can run for hours.

DoS HTTP/1.1 and above: Range Requests

By default max_ranges is not limited. DoS attacks can many Range-Requests (Impact on stability I/O).

Socket Sharding in NGINX 1.9.1+ (DragonFly BSD and Linux 3.9+)

Socket type Latency (ms) Latency stdev (ms) CPU Load
Default 15.65 26.59 0.3
accept_mutex off 15.59 26.48 10
reuseport 12.35 3.15 0.3

Thread Pools in NGINX Boost Performance 9x! (Linux)

Multi-threaded sending of files is currently supported only Linux. Without sendfile_max_chunk limit, one fast connection may seize the worker process entirely.

Happy Hacking!

nice document 👍

Sarfroz commented Aug 29, 2016

Not good for dedicated server

Owner

denji commented Aug 29, 2016 edited

Not good for dedicated server

@Sarfroz It is clear that this is not perfect for all tests, you can give more details? This is a test, this does not mean that all the settings were ideal for slow clients and production. If you read the comments there describes some of the parameters as not for production.

rahmatov commented Nov 9, 2016

Good tutorial ! Thanks !

Thanks!

Sarfroz commented Dec 18, 2016

sure sir my server details are below:
Intel Xeon E5
E5 1630v3 4c/8t
3.7 / 3.8 GHz 32 GB of RAM
DDR4 ECC 2133 MHz

Still after 5 users my server sleeps. If i use apache , it run smoothly whole day with 100 users without any problem. With nginx just one user connect and my server cpu load starts to increase. From last 8 months , i keep reading configuration of nginx , but trust me sir even google is now not able to find good working nginx config for me. I use my serve for leeching purpose. If you can help me i will be grateful to you. I have 6 servers all of them are on nginx , but none of them are well configured.

jun283 commented Dec 28, 2016

hello, can you consider below module:

https://github.com/cfsego/nginx-limit-upstream/

I think recommending tengine would be good and suitable here?

palaboy commented Feb 17, 2017 edited

@Sarfroz nice spec for you
what web application do you use that increase cpu load?
nginx & wordpress no problem for me, even wordpress.com uses nginx web server

@denji
boss how about for live production?

@yxliang01
tengine nginx version is old

Hi,

We use nodejs server. And use nginx to redirect request to upstream. We have 50K+ active connections at some point. What should be the value of keepalive in upstream to handle such many connections.

Currently we have 4 ec2 instance with -

upstream prodpass {
server 127.0.0.1:3001;
keepalive 2500;
}

But I guess with so many active connection at one time, keepalive value is less. Can you suggest how to decide keepalive parameter in upstream of nginx?

Why are all your values multiples of 10 and not 2? It's a little pedantic but buffer aligned values should be more efficient, no?

@joshenders How could something be a multiple of 10 and not 2? Do you mean powers? Would this actually help?

Sarfroz commented Aug 7, 2017

I am using the server for remote downloading files. So, is there any best configuration for such. Since a lot provider uses for this purpose and they are using it without any issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment