uWSGI and NGiNX

uwsgi-and-nginx.md

uWSGi and NGiNX Set-Up

Table of Contents generated with DocToc

• uWSGi and NGiNX Set-Up
  • Table of Contents
  • Brief introduction to each part
    • NGiNX
    • uWSGI
    • Celery
    • Flask
    • Virtualenv
    • Redis
  • How each component fits
  • Gimme the config files
    • /etc/uwsgi/emperor.ini
    • /etc/uwsgi/vassals-include.ini
    • /etc/uwsgi/vassals-inherit.ini
    • /etc/init/uwsgi.conf
    • /etc/nginx/sites-available/uwsgi.conf
    • /srv/default/uwsgi.app

Brief introduction to each part

NGiNX

The NGiNX Web Server will work as a reverse proxy and static asset server.

uWSGI

uWSGI will fulfill multiple roles:

• Application server, running the Python code in each of the vassals.
• Service monitor, starting and stopping vassals on-demand.
• Job manager: using the built-in daemon management helpers.
• App Router (called FastRouter).

Celery

Off-line task scheduler and worker. Celery will run the pipeline for building and distributing The Daily Williams.

Flask

The Web Framework, will serialize/deserialize the requests forwarded by uWSGI through NGiNX in the WSGI format.

Virtualenv

Virtualenv is a Python version manager that will provide a certain degree of isolation between installs.

Redis

Redis is a key-value database with Pub/Sub features.

How each component fits

NGiNX and the uWSGI Emperor instance get started by init (or another process, like supervisord) and do the basic setup: configurations get parsed, unix domain sockets get initialized, event loops start listening.

NGiNX works as a Reverse Proxy, meaning that it will receive all the incoming traffic and will delegate the requests (using the WSGI protocol) to the correct uWSGI App Server socket. NGiNX will also serve static files and wrap these requests using HTTPS.

The uWSGI Emperor Instance (hereon "Emperor") is the cornerstone of this setup. It will spawn, control, monitor and delegate requests to each uWSGI Application Server Vassal (the "Vassals"), and provides de appropiate execution isolation needed for each application.

The Emperor instance this setup uses also hosts the Application Router and Stats Server.

The Vassals are uWSGI instances that run in unprivileged mode and execute the application code. These nodes can be scaled either up or down, depending on the needs of the server. There is a 1:1 correspondence between the apps and vassals in this setup.

Celery is a task runner. It can be used to off-load long running tasks, build a highly concurrent pipeline and as a more advanced cron executor.

Redis is used as the broker and result database, the bridge between Celery and your Flask application.

Gimme the config files

/etc/uwsgi/emperor.ini

First, the Emperor:

[uwsgi]
uid = www-data
gid = www-data

exec-asap = mkdir -p /var/run/uwsgi
exec-asap = chown -R www-data:www-data /var/run/uwsgi

emperor = /srv/*/uwsgi.ini

vassals-include = /etc/uwsgi/vassals-include.ini
vassals-inherit = /etc/uwsgi/vassals-inherit.ini

touch-reload = /etc/uwsgi/emperor.ini
touch-reload = /etc/uwsgi/vassals-include.ini
touch-reload = /etc/uwsgi/vassals-inherit.ini

chmod-socket = 777
chown-socket = www-data

# Also work as fastrouter
shared-socket = /var/run/uwsgi/fastrouter.sock
fastrouter = =0
fastrouter-subscription-server = /var/run/uwsgi/fastrouter-sub.sock
fastrouter-fallback = /var/run/uwsgi/vassal-uwsgi-default.sock
fastrouter-fallback-on-no-key = true
fastrouter-workers = 2

stats = /var/run/uwsgi/emperor-stats.sock

auto-procname = true
vacuum = true
workers = 1
logto = /var/log/uwsgi/emperor.log
safe-pidfile = /var/run/uwsgi/emperor.pid

/etc/uwsgi/vassals-include.ini

The vassals (vassals-include.ini):

[uwsgi]
logto = /var/log/uwsgi/%N-%C.log
pidfile = /var/run/uwsgi/vassal-%C.pid

stats = /var/run/uwsgi/vassal-%C-stats.sock
chdir = %D

socket = /var/run/uwsgi/vassal-%C.sock

heartbeat = 10
max-requests = 100

subscribe-to = /var/run/uwsgi/fastrouter-sub.sock:%C

unsubscribe-on-graceful-reload = true

/etc/uwsgi/vassals-inherit.ini

And vassals-inherit.ini:

[uwsgi]
add-header = X-UA-Compatible: chrome=1
add-header = X-Sponsor: If you are reading this, consider donating to pcostesi.me!

/etc/init/uwsgi.conf

Now for something completely different, the uwsgi startup file:

# Emperor uWSGI script

description "uWSGI Emperor"
start on runlevel [2345]
stop on runlevel [06]
respawn

env UWSGI="/usr/local/bin/uwsgi"
exec "${UWSGI}" --ini "/etc/uwsgi/emperor.ini"

/etc/nginx/sites-available/uwsgi.conf

The NGiNX config file:

server {
	listen 80;
	listen 8000;
	server_name "~^(?<subdomain>.+)\.cloud\.pcostesi\.me$";
        access_log /var/log/nginx/uwsgi/access.log;
        error_log /var/log/nginx/uwsgi/error.log;
	client_max_body_size 10M;

	location /media/ {
		root /srv/$subdomain;
		autoindex	on;
	}

	location /static/ {
		root /srv/$subdomain;
		autoindex	on;
	}

	location / {
		uwsgi_pass unix:/var/run/uwsgi/fastrouter.sock;
		uwsgi_connect_timeout 6000;
		uwsgi_send_timeout 6000;
		uwsgi_read_timeout 6000;
		uwsgi_param UWSGI_FASTROUTER_KEY $subdomain;
		keepalive_timeout 6500;
		include uwsgi_params;
	}

	location ~ /\. {
		deny all;
	}
}

/srv/default/uwsgi.app

And the application config:

[uwsgi]
module = wsgi:app
pythonpath = %d
virtualenv = %d/env
chdir = %d

touch-reload = %d/uwsgi.ini
touch-reload = %d/uwsgi.app
touch-reload = %d/wsgi.py

harakiri = 10
max-requests = 5000
vacuum = true
process = 5

smart-attach-daemon = /tmp/celery/%C/celery.pid %(virtualenv)/bin/celery -A %d/tasks worker --pidfile=/tmp/celery/%C/celery.pid

[debug]
ini = :uwsgi
http = :8000
master = true