setting up a stack on webfaction

_webfaction_setup.rst

Setting up Webfaction for modern Python deployment

last updated: 4/5/2011

note that this stuff is always a moving target, much of this has been cribbed and combined from various blog posts. Much of the information was out of date from those, and if it is more than a couple months after the last updated date above, consider some of this likely to now be out of date.

Getting your basic python and home settings sorted out

Webfaction by default (at least on server I got) does not has virtualenv installed so we get virtualenv.py directly from it's github repo:

wget https://raw.github.com/pypa/virtualenv/master/virtualenv.py
python virtualenv.py $HOME

edit .bashrc and/or .bash_profile such that $HOME/bin and $HOME/sbin are on your path. From now on, your python will be from the virtualenv we just set up previously. Verify that by:

which python
~/bin/python

Now that we have our virtualenv properly set up in our $HOME, let's properly install virtualenv into our virtualenv. I know this sound so meta but I hope you can get along with it:

pip install --egg virtualenv
which virtualenv
~/bin/virtualenv

From now on to create new virtual environment, we'll just run virtualenv env_name.

This is optional but I prefer to have this. Let's create pip config file to define some default parameters:

mkdir -p ~/.pip/cache
cat - > ~/.pip/pip.conf
# ~/.pip/pip.conf
[global]
download_cache=/home/ptone/.pip/cache

# install as egg like easy_install did - new in pip-1.2.
egg = true
^D

This will tell pip to cache the download so we can save some bandwidth and speed up package installation.

nginx and uwsgi can be shared among projects and do not need to be rebuilt for each site:

cd ~
mkdir tmp src bin sbin etc conf mylogs sock

Download and make packages

http://nginx.org/en/download.html#stable_versions

cd src
curl http://nginx.org/download/nginx-0.8.54.tar.gz | tar -xvz

http://projects.unbit.it/uwsgi/wiki/RunOnNginx

grab upload progress module:

http://github.com/masterzen/nginx-upload-progress-module

git clone https://github.com/masterzen/nginx-upload-progress-module.git

get uwsgi itself: I've found this latest version has compile errors on webfaction while this does not:

curl http://projects.unbit.it/downloads/uwsgi-0.9.6.8.tar.gz | tar -xz

build uwsgi:

cd uwsgi-x-x-x-x
make -f Makefile.Py26
cp uwsgi $HOME/bin/

uwsgi module included by default in nginx since 0.8.40 so you do not need to specify it at configure time. Webfaction has sought to make the "logs" directory in your home folder owned by root - so we need to configure alternate log destinations.

build nginx:

cd ../nginx-0.8.54/
./configure --add-module=../nginx-upload-progress-module/ \
--prefix=$HOME \
--error-log-path=$HOME/mylogs/nginx-error.log \
--lock-path=$HOME/mylogs/nginx.lock \
--pid-path=$HOME/mylogs/nginx.pid \
--http-log-path=$HOME/mylogs/nginx-access.log
make
make install

expect to see errors about deprecated sys_errlist: http://nginx.org/en/docs/sys_errlist.html

Setting up the webfaction app

create custom port listening app:

cd webapps/<appname>/
virtualenv venv
source venv/bin/activate

Set up Django project

cd out to webapps/<appname>
django-admin.py startproject test_project

get the django project settings and urls set up for admin to work

collect static files:

manage.py collectstatic

Get the runtime stuff setup

get project on python path:

cd venv/<site-packages>
ln -s $HOME/webapps/<appname> .

start up nginx, note that it will not daemonize but can be started like this for testing:

nginx -p `pwd`/ -c ../test_project/etc/nginx.conf

create a wsgi.py file in project root:

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'test_project.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

startup uwsgi, again, won't daemonize - you'll need two terminals to run both nginx and uwsgi in this test mode:

uwsgi -p 4 -s $HOME/sock/teststack_uwsgi.sock -H $HOME/webapps/teststack/venv/ --pythonpath $HOME/webapps/teststack/test_project -w wsgi

You should now be able to load up the Django-admin

Running the setup with supervisor

create another custom port app in webfaction control panel

install supervisor with pip install supervisor

see supervisord.conf

see start_supervisor.sh

/10 * * * /home/ptone/bin/start_supervisor.sh start > /dev/null 2>&1 nohup start_supervisor.sh start

nginx.conf

#user  nobody;
worker_processes  2;

# pid        ptone-logs/nginx.pid;

# error_log  ptone-logs/nginx-error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;


# for running via supervisor:
daemon off;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  35;
    gzip  on;

    # Directories
    client_body_temp_path tmp/client_body/  2 2;
    fastcgi_temp_path     tmp/fastcgi/;
    proxy_temp_path       tmp/proxy/;
    uwsgi_temp_path       tmp/uwsgi/;

    # Logging
    access_log            log/nginx-access.log  combined;

# teststack server:
include /home/ptone/webapps/teststack/test_project/etc/nginx.conf;

    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443;
    #    server_name  localhost;

    #    ssl                  on;
    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_timeout  5m;

    #    ssl_protocols  SSLv2 SSLv3 TLSv1;
    #    ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
    #    ssl_prefer_server_ciphers   on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

pip.conf

# ~/.pip/pip.conf
[global]
download_cache=/home/ptone/.pip/cache

# install as egg like easy_install did - new in pip-1.2.
egg = true

project_nginx.conf

# Note that this file is located in a deploy or etc folder in the django project root

     # uWSGI serving Django.
    # probably not required in a non-loadbalancing scenario
    # upstream django {
        # Distribute requests to servers based on client IP. This keeps load
        # balancing fair but consistent per-client. In this instance we're
        # only using one uWGSI worker anyway.
        # ip_hash;
        # server unix:/home/ptone/sock/teststack_uwsgi.sock;
    # }

server {
        # teststack
        listen       50218;
        server_name  nginx.ptone.com;
        charset      utf-8;

        access_log  mylogs/teststack.access.log  combined;

        # Django admin media.
        location /static/admin/ {
        alias /home/ptone/webapps/teststack/venv/lib/python2.6/site-packages/django/contrib/admin/media/;
        }

        # Your project's static media.
        location /static/ {
        alias /home/ptone/webapps/teststack/test_project/media/;
        }

        # Finally, send all non-media requests to the Django server.
        location / {
        # uncomment below if you are using the upstream approach
        # uwsgi_pass  django;
        uwsgi_pass  unix:/home/ptone/sock/teststack_uwsgi.sock;
        include     uwsgi_params;
        }

    }

start_supervisor.sh

#! /bin/bash

NAME=supervisord
SUPERVISORD=/home/ptone/bin/supervisord
SUPERVISORCTL=/home/ptone/bin/supervisorctl
PIDFILE=/home/ptone/etc/supervisord.pid
OPTS="-c /home/ptone/etc/supervisord.conf"
PS=$NAME
TRUE=1
FALSE=0

test -x $SUPERVISORD || exit 0

export PATH="${PATH:+$PATH:}/usr/local/bin:/usr/sbin:/sbin:/home/ptone/bin:/home/ptone/sbin:"

isRunning(){
    pidof_daemon
    PID=$?

    if [ $PID -gt 0 ]; then
    return 1
    else
        return 0
    fi
}

pidof_daemon() {
    PIDS=`pidof -x $PS` || true

    [ -e $PIDFILE ] && PIDS2=`cat $PIDFILE`

    for i in $PIDS; do
        if [ "$i" = "$PIDS2" ]; then
            return 1
        fi
    done
    return 0
}

start () {
    echo "Starting Supervisor daemon manager..."
    isRunning
    isAlive=$?

    if [ "${isAlive}" -eq $TRUE ]; then
        echo "Supervisor is already running."
    else
        $SUPERVISORD $OPTS || echo "Failed...!"
        echo "OK"
    fi
}

stop () {
    echo "Stopping Supervisor daemon manager..."
    $SUPERVISORCTL shutdown ||  echo "Failed...!"
    echo "OK"
}

case "$1" in
  start)
    start
    ;;

  stop)
    stop
    ;;

  restart|reload|force-reload)
    stop
    start
    ;;

esac

exit 0

supervisord.conf

[inet_http_server]          ; inet (TCP) server setings
port=127.0.0.1:11672        ; (ip_address:port specifier, *:port for all iface)
username=<user>              ; (default is no username (open server))
password=<pass>              ; (default is no password (open server))

[supervisord]
logfile=/home/ptone/webapps/supervisor/logs/user/supervisor/supervisord.log ; (main log file                        ; default $CWD/supervisord.log)
logfile_maxbytes=20MB                                       ; (max main logfile bytes b4 rotation   ; default 50MB)
logfile_backups=10                                          ; (num of main logfile rotation backups ; default 10)
loglevel=debug                                              ; (log level                            ; default info                  ; others: debug,warn,trace)
pidfile=/home/ptone/etc/supervisord.pid                      ; (supervisord pidfile                  ; default supervisord.pid)
nodaemon=false                                              ; (start in foreground if true          ; default false)
minfds=1024                                                 ; (min. avail startup file descriptors  ; default 1024)
minprocs=200                                                ; (min. avail process descriptors       ; default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:11672
username=<user>
password=<pass>

[program:nginx]
command=nginx
autostart=true
autorestart=true
redirect_stderr=true
exitcodes=0

[program:teststack_uwsgi]
command=uwsgi -p 2 -s /home/ptone/sock/teststack_uwsgi.sock -H /home/ptone/webapps/teststack/venv/ --pythonpath /home/ptone/webapps/teststack/test_project -w wsgi
autostart=true
autorestart=true
redirect_stderr=true
exitcodes=0