Create a gist now

Instantly share code, notes, and snippets.

Zero downtime deploys with gunicorn
import multiprocessing
import os, os.path
import signal
# The Access log file to write to. [None]
accesslog='{}/logs/unicorn-access.log'.format(os.getcwd())
# The Access log format . [%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"]
# access_log_format='"%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
# Logging config file.
# logconfig="/opt/example/log.conf"
# The Error log file to write to. [-]
errorlog='{}/logs/unicorn-error.log'.format(os.getcwd())
# The granularity of Error log outputs. [info]
loglevel='debug'
# The logger you want to use to log events in gunicorn. [simple]
# logger_class='simple'
# A base to use with setproctitle for process naming. [None]
proc_name='your-app-name'
# Load application code before the worker processes are forked. [False]
preload_app=True
# forked. [False]
daemon=False
# daemon=True
# A filename to use for the PID file. [None]
# pidfile='/var/run/example.pid'
pidfile='{}/pids/unicorn.pid'.format(os.getcwd())
# Switch worker processes to run as this user. [0]
# user="example"
# Switch worker process to run as this group. [0]
# group="example"
# A bit mask for the file mode on files written by Gunicorn. [0]
umask=0002
# The socket to bind. [127.0.0.1:8000]
bind='0.0.0.0:8000'
# bind='unix:{}/sockets/unicorn.sock'.format(os.getcwd())
# The maximum number of pending connections. [2048]
# - Amazon Linux default=1024 ($ sysctl net.ipv4.tcp_max_syn_backlog)
backlog=2048
# The number of worker process for handling requests. [1]
workers=multiprocessing.cpu_count() * 2 + 1
# workers=1
# The type of workers to use. [sync]
worker_class='gevent'
# The maximum number of simultaneous clients. [1000]
worker_connections=4098
# The maximum number of requests a worker will process before restarting. [0]
max_requests=4098
# Workers silent for more than this many seconds are killed and restarted. [30]
timeout=120
#The number of seconds to wait for requests on a Keep-Alive connection. [2]
keepalive=2
# Timeout for graceful workers restart.
graceful_timeout=30
def pre_fork(server, worker):
old_pid_file = '{}/pids/unicorn.pid.oldbin'.format(os.getcwd())
if os.path.isfile(old_pid_file):
with open(old_pid_file, 'r') as pid_contents:
try:
old_pid = int(pid_contents.read())
if old_pid != server.pid:
os.kill(old_pid, signal.SIGQUIT)
except Exception as err:
pass
pre_fork=pre_fork
#!/bin/bash
#
# This file is managed by salt, using the unicorn formula template.
# Editing this file by hand is highly discouraged!
#
exec 2>&1
#
# Since unicorn creates a new pid on restart/reload, it needs a little extra
# love to manage with runit. Instead of managing unicorn directly, we simply
# trap signal calls to the service and redirect them to unicorn directly.
#
RUNIT_PID=$$
APPLICATION_NAME={{ unicorn_application }}
APPLICATION_PATH=/var/www/{{ unicorn_application }}/current
UNICORN_CMD={{ unicorn_command }}
CUR_PID_FILE=/var/www/{{ unicorn_application }}/shared/pids/unicorn.pid
OLD_PID_FILE=$CUR_PID_FILE.oldbin
echo "Runit service restarted (PID: $RUNIT_PID)"
function is_unicorn_alive {
set +e
if [ -n $1 ] && kill -0 $1 >/dev/null 2>&1; then
echo "yes"
fi
set -e
}
if [ -e $OLD_PID_FILE ]; then
OLD_PID=$(cat $OLD_PID_FILE)
echo "Old master detected (PID: $OLD_PID), waiting for it to quit"
while [ -n "$(is_unicorn_alive $OLD_PID)" ]; do
sleep 5
done
fi
if [ -e $CUR_PID_FILE ]; then
CUR_PID=$(cat $CUR_PID_FILE)
if [ -n "$(is_unicorn_alive $CUR_PID)" ]; then
echo "Detected running Unicorn instance (PID: $CUR_PID)"
RUNNING=true
fi
fi
function start {
unset ACTION
if [ $RUNNING ]; then
restart
else
echo 'Starting new unicorn instance'
RUNASUID=$(getent passwd {{ user }} | cut -d: -f3)
RUNASGROUPS=$(id -G {{ group }} | tr ' ' ':')
ulimit -n 65535
export HOME={{ home }}
export PATH={{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin:$PATH
source {{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin/activate
cd $APPLICATION_PATH
exec chpst -u :$RUNASUID:$RUNASGROUPS $UNICORN_CMD -c /var/www/{{ unicorn_application }}/current/config/gunicorn.py wsgi:app
sleep 3
CUR_PID=$(cat $CUR_PID_FILE)
fi
}
function stop {
unset ACTION
echo 'Initializing graceful shutdown'
kill -QUIT $CUR_PID
while [ -n "$(is_unicorn_alive $CUR_PID)" ]; do
echo '.'
sleep 2
done
echo 'Unicorn stopped, exiting Runit process'
kill -9 $RUNIT_PID
}
function restart {
unset ACTION
echo "Restart request captured, swapping old master (PID: $CUR_PID) for new master with USR2"
kill -USR2 $CUR_PID
sleep 2
echo 'Restarting Runit service to capture new master PID'
exit
}
function alarm {
unset ACTION
echo 'Unicorn process interrupted'
}
trap 'ACTION=stop' STOP TERM KILL
trap 'ACTION=restart' QUIT USR2 INT
trap 'ACTION=alarm' ALRM
[ $RUNNING ] || ACTION=start
if [ $ACTION ]; then
echo "Performing \"$ACTION\" action and going into sleep mode until new signal captured"
elif [ $RUNNING ]; then
echo "Going into sleep mode until new signal captured"
fi
if [ $ACTION ] || [ $RUNNING ]; then
while true; do
[ "$ACTION" == 'start' ] && start
[ "$ACTION" == 'stop' ] && stop
[ "$ACTION" == 'restart' ] && restart
[ "$ACTION" == 'alarm' ] && alarm
sleep 2
done
fi

Zero downtime deploys with gunicorn

Below are the actual files we use in one of our latest applications at Agora Games to achieve zero downtime deploys with gunicorn. I hope these files and notes help. I am happy to update these files or these notes if there are comments/questions. YMMV (of course).

Salient points for each file:

  • gunicorn.py: The pre_fork function looks for gunicorn's old PID file in the proper file and sends the proper QUIT signal to the old process once the new process is running.
  • sv-gunicorn-run.jinja: This is the runit template we use in our Salt-managed infrastructure for handling the application process management. You could just as easily convert this to a non-templatized version.
@dwickwire

I wonder if anyone got this working with a socket. When a USR2 is sent that spins up the oldbin just fine, but when you send the QUIT to it that kills the socket and the socket is never re-stablished with this method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment