Skip to content

Instantly share code, notes, and snippets.

@mikhailov
Last active June 29, 2024 23:43
Show Gist options
  • Save mikhailov/3052776 to your computer and use it in GitHub Desktop.
Save mikhailov/3052776 to your computer and use it in GitHub Desktop.
NGINX+SPDY with Unicorn. True Zero-Downtime unless migrations. Best practices.
# Nginx+Unicorn best-practices congifuration guide. Heartbleed fixed.
# We use latest stable nginx with fresh **openssl**, **zlib** and **pcre** dependencies.
# Some extra handy modules to use: --with-http_stub_status_module --with-http_gzip_static_module
#
# Deployment structure
#
# SERVER:
# /etc/init.d/nginx (1. nginx)
# /home/app/public_html/app_production/current (Capistrano directory)
#
# APP:
# config/server/production/nginx.conf (2. nginx.conf)
# config/server/production/nginx_host.conf (3. nginx_host.conf)
# config/server/production/nginx_errors.conf (4. nginx_errors.conf)
# config/deploy.rb (5. deploy.rb)
# config/deploy/production.rb (6. production.rb)
# config/server/production/unicorn.rb (7. unicorn.rb)
# config/server/production/unicorn_init.sh (8. unicorn_init.sh)
cd /usr/src
wget http://nginx.org/download/nginx-1.5.13.tar.gz
tar xzvf ./nginx-1.5.13.tar.gz && rm -f ./nginx-1.5.13.tar.gz
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.32.tar.gz
tar xzvf pcre-8.32.tar.gz && rm -f ./pcre-8.32.tar.gz
wget http://www.openssl.org/source/openssl-1.0.1g.tar.gz
tar xzvf openssl-1.0.1g.tar.gz && rm -f openssl-1.0.1g.tar.gz
cd nginx-1.5.13
./configure --prefix=/opt/nginx --with-pcre=/usr/src/pcre-8.32 --with-openssl-opt=no-krb5 --with-openssl=/usr/src/openssl-1.0.1g --with-http_ssl_module --with-http_spdy_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --with-http_stub_status_module --with-http_gzip_static_module
make && make install
mkdir /tmp/client_body_temp
mkdir /opt/nginx/ssl_certs
echo "include /home/app/public_html/app_production/current/config/server/production/nginx.conf;" > /opt/nginx/conf/nginx.conf
vim /etc/init.d/nginx # see the config below
chmod +x /etc/init.d/nginx && update-rc.d -f nginx defaults
#! /bin/sh
### BEGIN INIT INFO
# Provides: nginx
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the nginx web server
# Description: starts nginx using start-stop-daemon
### END INIT INFO
#
# /etc/init.d/nginx
PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx
test -x $DAEMON || exit 0
# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
. /etc/default/nginx
fi
set -e
case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon --start --quiet --pidfile /opt/nginx/logs/$NAME.pid \
--exec $DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --pidfile /opt/nginx/logs/$NAME.pid \
--exec $DAEMON
echo "$NAME."
;;
restart|force-reload)
echo -n "Restarting $DESC: "
start-stop-daemon --stop --quiet --pidfile \
/opt/nginx/logs/$NAME.pid --exec $DAEMON
sleep 1
start-stop-daemon --start --quiet --pidfile \
/opt/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
reload)
echo -n "Reloading $DESC configuration: "
start-stop-daemon --stop --signal HUP --quiet --pidfile /opt/nginx/logs/$NAME.pid \
--exec $DAEMON
echo "$NAME."
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
exit 1
;;
esac
exit 0
# Nginx main block configuration file.
# The most important directives here are ssl_protocols and ssl_ciphers
# Keep nginx configuration in repo, then just include it in /opt/nginx/conf/nginx.conf
# And DDoS prevent attack with directive limit_req_zone (limit 10 request/sec from 1 IP address)
# then it enables in the server block by "limit_req zone=one".
#
# config/server/production/nginx.conf
user app;
worker_processes 2;
worker_priority -5;
timer_resolution 100ms;
error_log logs/nginx.error.log;
events {
use epoll;
worker_connections 2048;
}
http {
client_max_body_size 25m;
client_body_buffer_size 128k;
client_body_temp_path /tmp/client_body_temp;
include mime.types;
default_type application/octet-stream;
server_tokens off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 70;
gzip on;
gzip_http_version 1.1;
gzip_disable "msie6";
gzip_vary on;
gzip_min_length 1100;
gzip_buffers 64 8k;
gzip_comp_level 3;
gzip_proxied any;
gzip_types text/plain text/css application/x-javascript text/xml application/xml;
ssl_certificate /opt/nginx/ssl_certs/server.crt;
ssl_certificate_key /opt/nginx/ssl_certs/server.key;
ssl_session_timeout 15m;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_stapling on;
add_header Strict-Transport-Security "max-age=16070400; includeSubdomains";
add_header X-Frame-Options DENY;
limit_req_zone $binary_remote_addr zone=one:10m rate=50r/s;
include /home/app/public_html/app_production/current/config/server/nginx_host.conf;
}
# Nginx server block configuration with proxy_pass to Unicorn upstream
# We use full-SSL site with web-server redirection, no mess with Rails application redirection
#
# config/server/production/nginx_host.conf
upstream unicorn {
server unix:/tmp/unicorn.production.sock fail_timeout=0;
}
server {
listen 80;
server_name server.com;
rewrite ^(.*) https://$host$1 permanent;
location ~ \.(php|html)$ {
deny all;
}
access_log /dev/null;
error_log /dev/null;
}
server {
ssl on;
listen 443 spdy ssl;
server_name server.com;
root /home/app/public_html/app_production/current/public;
try_files $uri /system/maintenance.html @unicorn;
location @unicorn {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://unicorn;
limit_req zone=one burst=15;
access_log /dev/null;
error_log logs/unicorn.error.log;
}
location ~ ^/(assets|images|javascripts|stylesheets|swfs|system)/ {
gzip_static on;
expires max;
add_header Cache-Control public;
add_header Last-Modified "";
add_header ETag "";
}
include /home/app/public_html/app_production/current/config/server/production/nginx_errors.conf;
access_log /dev/null;
error_log /dev/null;
}
# nginx configuration piece to handle errorrs
#
# config/server/production/nginx_errors.conf
error_page 500 502 504 /500.html;
error_page 503 @503;
location = /50x.html {
root html;
}
location = /404.html {
root html;
}
location @503 {
error_page 405 = /system/maintenance.html;
if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html break;
}
rewrite ^(.*)$ /503.html break;
}
if ($request_method !~ ^(GET|HEAD|PUT|POST|DELETE|OPTIONS)$ ){
return 405;
}
if (-f $document_root/system/maintenance.html) {
return 503;
}
location ~ \.(php|html)$ {
return 405;
}
# Capistrano configuration. Now with TRUE zero-downtime unless DB migration.
#
# require 'new_relic/recipes' - Newrelic notification about deployment
# require 'capistrano/ext/multistage' - We use 2 deployment environment: staging and production.
# set :deploy_via, :remote_cache - fetch only latest changes during deployment
# set :normalize_asset_timestamps - no need to touch (date modification) every assets
# "deploy:web:disable" - traditional maintenance page (during DB migrations deployment)
# task :restart - Unicorn with preload_app should be reloaded by USR2+QUIT signals, not HUP
#
# http://unicorn.bogomips.org/SIGNALS.html
# "If “preload_app” is true, then application code changes will have no effect;
# USR2 + QUIT (see below) must be used to load newer code in this case"
#
# config/deploy.rb
require 'bundler/capistrano'
require 'capistrano/ext/multistage'
require 'new_relic/recipes'
set :stages, %w(staging production)
set :default_stage, "staging"
set :scm, :git
set :repository, "..."
set :deploy_via, :remote_cache
default_run_options[:pty] = true
set :application, "app"
set :use_sudo, false
set :user, "app"
set :normalize_asset_timestamps, false
before "deploy", "deploy:delayed_job:stop"
before "deploy:migrations", "deploy:delayed_job:stop"
after "deploy:update_code", "deploy:symlink_shared"
before "deploy:migrate", "deploy:web:disable", "deploy:db:backup"
after "deploy", "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"
after "deploy:migrations", "deploy:web:enable", "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"
namespace :deploy do
%w[start stop].each do |command|
desc "#{command} unicorn server"
task command, :roles => :app, :except => { :no_release => true } do
run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh #{command}"
end
end
desc "restart unicorn server"
task :restart, :roles => :app, :except => { :no_release => true } do
run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh upgrade"
end
desc "Link in the production database.yml and assets"
task :symlink_shared do
run "ln -nfs #{deploy_to}/shared/config/database.yml #{release_path}/config/database.yml"
end
namespace :delayed_job do
desc "Restart the delayed_job process"
task :restart, :roles => :app, :except => { :no_release => true } do
run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job restart" rescue nil
end
desc "Stop the delayed_job process"
task :stop, :roles => :app, :except => { :no_release => true } do
run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job stop" rescue nil
end
end
namespace :db do
desc "backup of database before migrations are invoked"
task :backup, :roles => :db, :only => { :primary => true } do
filename = "#{deploy_to}/shared/db_backup/#{stage}_db.#{Time.now.utc.strftime("%Y-%m-%d_%I:%M")}_before_deploy.gz"
text = capture "cat #{deploy_to}/current/config/database.yml"
yaml = YAML::load(text)["#{stage}"]
on_rollback { run "rm #{filename}" }
run "mysqldump --single-transaction --quick -u#{yaml['username']} -h#{yaml['host']} -p#{yaml['password']} #{yaml['database']} | gzip -c > #{filename}"
end
end
namespace :web do
desc "Maintenance start"
task :disable, :roles => :web do
on_rollback { run "rm #{shared_path}/system/maintenance.html" }
page = File.read("public/503.html")
put page, "#{shared_path}/system/maintenance.html", :mode => 0644
end
desc "Maintenance stop"
task :enable, :roles => :web do
run "rm #{shared_path}/system/maintenance.html"
end
end
end
namespace :log do
desc "A pinch of tail"
task :tailf, :roles => :app do
run "tail -n 10000 -f #{shared_path}/log/#{rails_env}.log" do |channel, stream, data|
puts "#{data}"
break if stream == :err
end
end
end
# capistrano production config
#
# config/deploy/production.rb
server "8.8.8.8", :app, :web, :db, :primary => true
set :branch, "production"
set :deploy_to, "/home/app/public_html/app_production"
set :rails_env, "production"
# Unicorn configuration file to be running by unicorn_init.sh with capistrano task
# read an example configuration before: http://unicorn.bogomips.org/examples/unicorn.conf.rb
#
# working_directory, pid, paths - internal Unicorn variables must to setup
# worker_process 4 - is good enough for serve small production application
# timeout 30 - time limit when unresponded workers to restart
# preload_app true - the most interesting option that confuse a lot of us,
# just setup is as true always, it means extra work on
# deployment scripts to make it correctly
# BUNDLE_GEMFILE - make Gemfile accessible with new master
# before_fork, after_fork - reconnect to all dependent services: DB, Redis, Sphinx etc.
# deal with old_pid only if CPU or RAM are limited enough
#
# config/server/production/unicorn.rb
app_path = "/home/app/public_html/app_production/current"
working_directory "#{app_path}"
pid "#{app_path}/tmp/pids/unicorn.pid"
stderr_path "#{app_path}/log/unicorn.log"
stdout_path "#{app_path}/log/unicorn.log"
listen "/tmp/unicorn.production.sock"
worker_processes 4
timeout 30
preload_app true
before_exec do |server|
ENV["BUNDLE_GEMFILE"] = "#{app_path}/Gemfile"
end
before_fork do |server, worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
end
if defined?(Resque)
Resque.redis.quit
end
sleep 1
end
after_fork do |server, worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
end
if defined?(Resque)
Resque.redis = 'localhost:6379'
end
end
# Unicorn handle shell script
#
# APP_ROOT, PID - are the same as you setup above
# CMD - use bundle binstubs (bundle install --binstubs) to
# forget about "bundle exec" stuff, run in demonize mode
# bin/unicorn is for Rack application (config.ru in root dir), but
# bin/unicorn_rails is to use with Rails 2.3
#
# To handle "app_preload true" configuration we should use USR2+QUIT signals, not HUP!
# So we rewrite capistrano deployment scripts to manage it.
#
# config/server/production/unicorn_init.sh
#!/bin/sh
set -e
# Example init script, this can be used with nginx, too,
# since nginx and unicorn accept the same signals
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/app/public_html/app_production/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="$APP_ROOT/bin/unicorn -D -c $APP_ROOT/config/server/unicorn.rb -E production"
action="$1"
set -u
old_pid="$PID.oldbin"
cd $APP_ROOT || exit 1
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $old_pid && kill -$1 `cat $old_pid`
}
case $action in
start)
sig 0 && echo >&2 "Already running" && exit 0
$CMD
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
sig HUP && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
$CMD
;;
upgrade)
if sig USR2 && sleep 15 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $old_pid && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $old_pid
then
echo >&2 "$old_pid still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
$CMD
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
@fillman
Copy link

fillman commented Jul 9, 2012

отличный конфиг, букмаркаю, nginx + unicorn, шустрая тема

@mikhailov
Copy link
Author

UPDATE Nginx caching open file descriptors (static assets only), their sizes and modification times with using of open_file_cache

@mikhailov
Copy link
Author

UPDATE unicorn USR2 with followed by sleep for 15sec of 2. This time is enough to preload the application.

@alfuken
Copy link

alfuken commented Sep 13, 2012

при выполнении upgrade поулчаю следующее:

Couldn't upgrade, starting '/home/vagrant/app/current/bin/unicorn_rails -D -c /home/vagrant/app/shared/config/unicorn.rb -E production' instead
master failed to start, check stderr log for details

В логах следующее:

I, [2012-09-12T15:31:02.127414 #1417]  INFO -- : executing ["/home/vagrant/app/current/bin/unicorn_rails", "-D", "-c", "/home/vagrant/app/shared/config/unicorn.rb", "-E", "production", {5=>#<Kgio::UNIXServer:/home/vagrant/app/shared/tmp/sockets/unicorn.sock>}] (in /home/vagrant/app/releases/1)
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- bundler/setup (LoadError)
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /home/vagrant/app/current/bin/unicorn_rails:14:in `<main>'

В чем может быть проблема?

@mikhailov
Copy link
Author

@alfuken gem install bundler и проверь, что свежий Gemfile.lock в репо

@mikhailov
Copy link
Author

UPDATE: now with SPDY support

@xymbol
Copy link

xymbol commented Nov 3, 2012

@mikhailov, any reason for running Unicorn processes as root? Thanks for sharing.

@mikhailov
Copy link
Author

@xymbol master process runs as root, but workers as app

@BrindleFly
Copy link

The init.d scripts start nginx and unicorn but cannot perform the other actions (e.g. stop). Not sure if this is Ubuntu specific or not, but start-stop-daemon will not create a pid file without the flag --make-pid - and even then will write the wrong pid due to the nginx/unicorn processes forking from the main process.

@k3NGuru
Copy link

k3NGuru commented Mar 26, 2013

Здравствуйте. Сделал как написано в вашем конфиге, но у меня не стартуют рельсы.
В логах nginx все норм, а вот в логах unicorn.error.log данное сообщение

2013/03/26 15:49:41 [error] 25371#0: *8 limiting requests, excess: 1.000 by zone "one", client: 192.168.91.185, server: localhost, request: "GET / HTTP/1.1", host: "192.168.91.15"

А сама страница отдает 404 Ошибку

@juniorjp
Copy link

Very good. Please change the comment: (limit 10 request/sec from 1 IP address) to (limit 50...) about nginx limit_req_zone.

@mikhailov
Copy link
Author

@k3NGuru по-моему, limit_req_zone был поломан и его починили только несколько месяцев назад, пробуйте стартануть nginx без него

@k3NGuru
Copy link

k3NGuru commented Apr 5, 2013

@mikhailov Все равно не заводится система. Уже сделал как написано тут https://coderwall.com/p/8igwqa настроил как написано в данном конфиге, Вот моя папка https://github.com/k3NGuru/mlvz

2013/04/05 16:02:07 [error] 19467#0: *2 limiting requests, excess: 1.000 by zone "one", client: 192.168.91.163, server: 192.168.91.15, request: "GET / HTTP/1.1^@^Dho^@^@", host: "192.168.91.15"

192.168.91.15 - Сервер
192.168.91.163 - Клиент

SSL настроил как написано тут http://wiki.nginx.org/HttpSslModule

@mikhailov
Copy link
Author

@k3NGuru все дело в специфичной работе встроенного модуля limit_req, добавь burst=5 для решения проблемы:

limit_req         zone=one burst=5;

Это пропустит 6 запросов в секунду, надо больше с одной айпи - увеличивай burst, либо распараллель нагрузку между несколькими Nginx серверами. Почему так, а не иначе, попытались объяснить здесь: http://www.lexa.ru/nginx-ru/msg38569.html Сегодня задан вопрос в nginx-рассылку.

@klebershimabuku
Copy link

Thank you for sharing this. :-)

@rusllonrails
Copy link

Amazing. Thank you very much man =)

@collinsethans
Copy link

Great stuff to ease nginx usage!

A few queries/points:

  1. I needed to add ' application/javascript' to gzip_types, else js was not getting compressed. Added that before application/x-javascript. Any reason why you don't have that?
  2. keepalive_timeout of 70 seems to be a very very high value. Isn't it better to control it within 2/3 (as otherwise it's easy for hackers to drown the server with open files)

@cinic
Copy link

cinic commented Mar 26, 2015

require 'capistrano/ext/multistage'

Данное расширение объединено с основной веткой Capistrano capistrano/capistrano@eed1a6e. Можно исключить из конфига.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment