Serving Django and Twisted using HAproxy

celery.conf

[program:YOUR_PROJECT_celery]
command=python manage.py celeryd --loglevel=INFO --beat
environment=PATH="/home/www-data/YOUR-PROJECT/dj/env/bin:/usr/local/bin:/usr/bin:/bin"
directory=/home/www-data/YOUR-PROJECT/dj/
user=www-data
numprocs=1
umask=022
stdout_logfile=/home/www-data/logs/%(program_name)s.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/home/www-data/logs/%(program_name)s.error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
autorestart=True
autostart=True
startsecs=10
stopwaitsecs = 60
priority=998

DJANGO_TWISTED_HAPROXY.md

Serving Django and Twisted using HAproxy

Why?

Because we wanted to achive the following while developing a webapp using websockets:

Static serving (nginx), Django application (gunicorn) and Websockets service (twisted)

• on the same IP

  because small vhosts come with one address only.

• on the same Port 80

  to support companies which are allowing connections to common ports only.

Who?

This setup was mostly created by @frog32 and the guide was written by @sspross. It was developed while working at allink. Warning: This setup is not battle tested and may or may not work for you.

This demo application by @frog32 is using this setup, but is still under havy development.

How?

0. Project Setup
1. HAproxy Setup
2. Gunicorn Setup
3. Nginx Setup
4. Twisted Setup
5. Celery Setup

Django and Twisted

• Our project structure looks like this:

  • /home/www-data/YOUR-PROJECT/dj/ <- Django Project
  • /home/www-data/YOUR-PROJECT/tx/ <- Twisted Project

• Installed packages:

  apt-get install csstidy yui-compressor libpng12-dev libjpeg-dev zlib1g-dev python-virtualenv  python-setuptools  libxml2-dev git gettext python-dev libssl-dev mercurial libmysqlclient-dev redis-server rabbitmq-server mysql-server haproxy supervisor nginx node-less
  

Project Setup

• Python packages (e.g. REQUIREMENTS_SERVER):

  hiredis
  redis
  django-redis-cache
  gunicorn
  

• Add to your settings.py config:

  USE_X_FORWARDED_HOST = True
  

• Provide URL in urls.py for HAproxy to enable monitoring:

  url(r'^isrunning$', lambda r: HttpResponse('OK')),
  

HAproxy Setup

• Config HAproxy /etc/haproxy/haproxy.cfg, e.g. this.

• Enable HAproxy, set ENABLED=1 in /etc/default/haproxy (and maybe in /etc/init.d/haproxy) and start it with sudo /etc/init.d/haproxy start. You should see the following if it's really running:

   * Starting haproxy haproxy
     ...done.
  

• If your firewall is active, open port for HAproxy stats sudo ufw allow 1936

• Now you can access HAproxy stats over http://YOUR.IP:1936/ and gunicorn should be up and working.

Gunicorn Setup

• There has to be a wsgi.py file in /home/www-data/YOUR-PROJECT/dj/, e.g. this.
• Create supervisor configuration for gunicorn /etc/supervisor/conf.d/YOUR_PROJECT_gunicorn.conf, e.g. this.
• Don't forget to create a logs directory as configured in the gunicorn config.
• Update supervisor sudo supervisorctl run update (to restart webserver use restart YOUR_PROJECT_gunicorn)

Nginx Setup

• Unlink default site sudo unlink /etc/nginx/sites-enabled/default
• Create an enabled site configuration /etc/nginx/sites-enabled/YOUR-DOMAIN.ch, e.g. this.
• Don't forget to create a isrunning file like configured e.g. touch /home/www-data/YOUR-PROJECT/dj/isrunning for HAproxy Ping.
• Restart nginx sudo /etc/init.d/nginx restart
• Now check HAproxy stats if nginx is pingable and up running http://YOUR.IP:1936/

Twisted Setup

• Create an additional virtualenv for your Twisted app in /home/www-data/YOUR-PROJECT/tx/ with virtualenv --setuptools env

• Install python packages:

  txsockjs
  pyopenssl
  twisted
  

• Create another supervisor configuration for twisted /etc/supervisor/conf.d/YOUR_PROJECT_twisted.conf, e.g. this.

• Update supervisor sudo supervisorctl run update (to restart twisted use restart YOUR_PROJECT_twisted)

• Now both (gunicorn and twisted) should be running:

  supervisor> status
  YOUR_PROJECT_gunicorn             RUNNING    pid 11194, uptime 0:03:34
  YOUR_PROJECT_twisted              RUNNING    pid 11231, uptime 0:01:57
  

• And everything should be green on the HAproxy status page http://YOUR.IP:1936/.

Celery Setup

• Add to your settings.py file:

  # broker and celery
  BROKER_URL = 'amqp://YOUR_PROJECT:YOUR_PROJECT@localhost:5672/YOUR_PROJECT'
  CELERY_RESULT_BACKEND = "redis://localhost/0"
  CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
  CELERYD_CONCURRENCY = 1
  CELERY_SEND_EVENTS = False
  CELERY_ENABLE_UTC = True
  
  import djcelery
  djcelery.setup_loader()
  

• Create rabbitmq user and vhost:

  sudo rabbitmqctl add_user YOUR_PROJECT YOUR_PROJECT
  sudo rabbitmqctl add_vhost YOUR_PROJECT
  sudo rabbitmqctl set_permissions -p YOUR_PROJECT YOUR_PROJECT ".*" ".*" ".*"
  

• Create another supervisor configuration for celery /etc/supervisor/conf.d/YOUR_PROJECT_celery.conf, e.g. this.

• Update supervisor sudo supervisorctl run update (to restart celery use restart YOUR_PROJECT_celery)

• Now all three (gunicorn, twisted and celery) should be running:

  supervisor> status
  YOUR_PROJECT_celery               RUNNING    pid 12281, uptime 0:00:13
  YOUR_PROJECT_gunicorn             RUNNING    pid 11194, uptime 1:18:23
  YOUR_PROJECT_twisted              RUNNING    pid 11231, uptime 1:16:46
  

gunicorn.conf

[program:YOUR_PROJECT_gunicorn]
command=/home/www-data/YOUR-PROJECT/dj/env/bin/gunicorn -b 127.0.0.1:9001 wsgi:application
directory=/home/www-data/YOUR-PROJECT/dj
user=www-data
numprocs=1
umask=022
stdout_logfile=/home/www-data/logs/%(program_name)s.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/home/www-data/logs/%(program_name)s.error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
autorestart=true
autostart=True
startsecs=10
stopwaitsecs = 60
priority=998

haproxy.cfg

global
    log 127.0.0.1	local0
    log 127.0.0.1	local1 notice
    #log loghost	local0 info
    maxconn 4096
    #chroot /usr/share/haproxy
    user haproxy
    group haproxy
    daemon
    #debug
    #quiet

defaults
    log	global
    mode	http
    option	httplog
    option	dontlognull
    retries	3
    option redispatch
    # reevaluate rules for every request
    option http-server-close
    # the next line is needed to not fall back to streaming
    option http-pretend-keepalive
    maxconn	2000
    contimeout	5000
    clitimeout	50000
    srvtimeout	50000
    option forwardfor

frontend public
    bind :80
  
    timeout client 86400000
    
    acl is_websocket path_beg /ws
    #acl is_websocket hdr(Upgrade) -i WebSocket
    acl url_static    path_beg    /static/ /media/
    use_backend twisted if is_websocket
    
    use_backend nginx if url_static
    default_backend gunicorn

backend twisted
    # put the same clients allways on the same server
    balance uri depth 2
    option httpchk GET /ws/
    timeout check 1000ms
    #reqrep ^([^\ ]*)\ /ws/(.*) \1\ /\2
    server twisted1 127.0.0.1:10001 check inter 5s
    #server twisted2 127.0.0.1:10002 check inter 1000ms

backend nginx
    balance leastconn
    option httpchk GET /isrunning
    timeout check 500ms
    server nginx1 127.0.0.1:8001 check inter 1s
    #server nginx2 127.0.0.1:8002 check inter 1000ms

backend gunicorn
    balance leastconn
    option httpchk GET /isrunning
    timeout check 500ms
    server gunicorn1 127.0.0.1:9001 check inter 5s 

frontend stats
    bind :1936
    default_backend stats

backend stats
    stats enable
    stats hide-version
    stats realm HAproxy\Statistics
    stats uri /
    stats auth YOUR_HAPROXY_USERNAME:YOUR_HAPROXY_PASSWORD
    
    rspidel ^Set-cookie:\ IP=	# do not let this cookie tell our internal IP address
    
    #errorloc	502	http://192.168.114.58/error502.html
    #errorfile	503	/etc/haproxy/errors/503.http
    errorfile	400	/etc/haproxy/errors/400.http
    errorfile	403	/etc/haproxy/errors/403.http
    errorfile	408	/etc/haproxy/errors/408.http
    errorfile	500	/etc/haproxy/errors/500.http
    errorfile	502	/etc/haproxy/errors/502.http
    errorfile	503	/etc/haproxy/errors/503.http
    errorfile	504	/etc/haproxy/errors/504.http

ngnix-enabled-site.txt

server {
    listen   8001;
    root /home/www-data/YOUR-PROJECT/dj/;
    location /media/ {
    }
    location /static/ {
    }
    location /isrunning {
    }
    location / {
        deny all;
    }
    # what to serve if upstream is not available or crashes
    error_page 500 502 503 504 /media/50x.html;
}

SERVER_SETUP.md

In this guide we used a digitalocean droplet, but it is just an Ubuntu Server and not for production usage!

0. Enable Automatic Security Updates
1. Install Logwatch To Keep An Eye On Things
2. Set Up a Firewall
3. Switch to www-data User and require Public Key Authentication
4. Lock Down SSH
5. Colors for www-data

Ubuntu Server

• Add your SSH key to your digital ocean control panel
• Create a droplet (Ubuntu 12.04 x32 Server)
• Login over ssh -A root@YOUR.IP
• Change root password passwd (you don't have to remember it, you login with your ssh key and you can reset the root password using the control panel anytime)
• Update distribution apt-get update && apt-get upgrade
• Install fail2ban apt-get install fail2ban

Enable Automatic Security Updates

• apt-get install vim unattended-upgrades and edit vim /etc/apt/apt.conf.d/10periodic:

  APT::Periodic::Update-Package-Lists "1";
  APT::Periodic::Download-Upgradeable-Packages "1";
  APT::Periodic::AutocleanInterval "7";
  APT::Periodic::Unattended-Upgrade "1";
  

Install Logwatch To Keep An Eye On Things

• apt-get install logwatch and edit vim /etc/cron.daily/00logwatch:

  /usr/sbin/logwatch --output mail --mailto YOUR@EMAIL.COM --detail high
  

Set Up a Firewall

• apt-get install ufw and

  ufw allow 22
  ufw allow 80
  ufw enable
  

  (allow only from a specific IP ufw allow from {your-ip} to any port 22, allow https ufw allow 443

Switch to www-data User and require Public Key Authentication

• Change home folder for user www-data and add your public key to the authorized_keys file:

  mkdir /home/www-data
  mkdir /home/www-data/.ssh
  chmod 700 /home/www-data/.ssh
  vim /home/www-data/.ssh/authorized_keys
  chown www-data:www-data /home/www-data -R
  usermod -d /home/www-data www-data
  

• Activate sudo for www-data:

  passwd www-data
  

• In visudo comment all existing user/group grant lines and add:

  root      ALL=(ALL) ALL
  www-data  ALL=(ALL) ALL
  

  Maybe allow www-data to use supervisor without password:

  www-data  ALL=(ALL) NOPASSWD: /usr/bin/supervisorctl
  

Lock Down SSH

• Configure ssh to prevent password & root logins vim /etc/ssh/sshd_config:

  PermitRootLogin no
  PasswordAuthentication no
  

• And restart ssh service ssh restart

Colors for www-data

• Edit /etc/passwd and change the shell of www-data user from /bin/sh to /bin/bash
• Copy bashrc and profile files cp /root/.bashrc /home/www-data/ && cp /root/.profile /home/www-data/ and update their owner chown www-data:www-data /home/www-data -R

References

• [1] http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers

twisted.conf

[program:YOUR_PROJECT_twisted]
command=/home/www-data/YOUR-PROJECT/tx/env/bin/twistd -n -y service.tac
directory=/home/www-data/YOUR-PROJECT/tx
user=www-data
numprocs=1
umask=022
stdout_logfile=/home/www-data/logs/%(program_name)s.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/home/www-data/logs/%(program_name)s.error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
autorestart=true
autostart=True
startsecs=10
stopwaitsecs = 60
priority=998

wsgi.py

import os

PROJECT_NAME = 'YOUR_PROJECT'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "%s.settings" % PROJECT_NAME)

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()