with Fab 0.9 and hg & TG2 instead of git & Django

gistfile1

"""

NOTES: 
  * Originally from http://morethanseven.net/2009/07/27/fabric-django-git-apache-mod_wsgi-virtualenv-and-p/
  * Patches for fabric 0.9/1.0 with code from <http://github.com/fiee/generic_django_project/>

Changes from the original fabfile:
    * Patches for Fabric 0.9/1.0
    * Use Mercurial instead of Git
    * Use named mercurial branches instead of head/tip
    * Allow multiple deployment environments: imagine "alpha" vs "beta" vs "live"
    * Fixed bug: now looks for current and previous: only deletes if there
    * Allow virtualenv location to be specified
    [TODO] * Use modwsgideploy directory structure
    * Use Turbogears instead of Django
    * Since it's Turbogears, run paster setup-app production.ini
        (you are responsible for making this run correctly when it is run repeatedly - 
        for example, having it load up your database migrations and various bootstrap
        data for you. Sample projects exist to show you how to do this,
        for example <http://github.com/rwilcox/turbogears2_boostrap_data/>)

On using named branches instead of head/tip
------------------------------------
Create an archive from the most current branch that matches the environment you are deploying.
For example, if you are deploying to the test environment, will look for
the most recent hg branch that begins with test.

This allows us to be able to tag branches for multiple test environments,
and deploy them separately. It also allows you to tag a commit to be deployed
to a server even after you've made many commits after that one (commits you do
NOT want deployed).

Spaces are not allowed in the tag name.

We use tags instead of branches because they work for the 80 percent scenario:
a deploy is fine, there are no fixes needed, and the fixes that are needed can
wait until the next deploy.

If you have to make a series of immediate bug fixes, you can hg update to the
tag you deployed, branch at that point, then retag for deployment.

Allow multiple deployment environments: imagine "alpha" vs "beta" vs "live"
------------------------------------

For your different environments, modify alpha_webserver, beta_webserver, and production_webserver
as appropriate.

On modwsgideploy:
------------------------------------
    We also assume you are using modwsgideploy (<http://pypi.python.org/pypi/modwsgideploy>)
    to create the apache config. This just gives us a standard place to put/get
    the modwsgi configurations. We'll use the generated folder structure with
    some modification: the generated apache folder should be postfixed with the
    environment name.
    
    For example, for the alpha deployment environment:
    apache_alpha
    
    And for the production deployment environment:
    apache_production

This fabfile also assumes you are deploying on a very Linux like Apache2 setup.
If you're running on Mac OS X (Client, at least) you won't have a site-available
folder in your /etc/apache2/. What a shame - I think you should have one.

So make one, then add this line to your http.conf:

    Include /private/etc/apache2/sites-available/*.conf

I disagree with the authors of modwsgideploy, in that the Apache config file
should have a .config extension. Please add it.

From the original file
=============================================================================
This fabric file makes setting up and deploying a django application much
easier, but it does make a few assumptions. Namely that you're using Git,
Apache and mod_wsgi and your using Debian or Ubuntu. Also you should have 
Django installed on your local machine and SSH installed on both the local
machine and any servers you want to deploy to.

_note that I've used the name project_name throughout this example. Replace
this with whatever your project is called._

First step is to create your project locally:

    mkdir project_name
    cd project_name
    django-admin.py startproject project_name

Now add a requirements file so pip knows to install Django. You'll probably
add other required modules in here later. Creat a file called requirements.txt
and save it at the top level with the following contents:

    Django
    
Then save this fabfile.py file in the top level directory which should give you:
    
    project_name
        fabfile.py
        requirements.txt
        project_name
            __init__.py
            manage.py
            settings.py
            urls.py

You'll need a WSGI file called project_name.wsgi, where project_name 
is the name you gave to your django project. It will probably look 
like the following, depending on your specific paths and the location
of your settings module

    import os
    import sys

    # put the Django project on sys.path
    sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../")))

    os.environ["DJANGO_SETTINGS_MODULE"] = "project_name.settings"

    from django.core.handlers.wsgi import WSGIHandler
    application = WSGIHandler()

Last but not least you'll want a virtualhost file for apache which looks 
something like the following. Save this as project_name in the inner directory.
You'll want to change /path/to/project_name/ to the location on the remote
server you intent to deploy to.

    <VirtualHost *:80>
        WSGIDaemonProcess project_name-production user=project_name group=project_name threads=10 python-path=/path/to/project_name/lib/python2.6/site-packages
        WSGIProcessGroup project_name-production

        WSGIScriptAlias / /path/to/project_name/releases/current/project_name/project_name.wsgi
        <Directory /path/to/project_name/releases/current/project_name>
            Order deny,allow
            Allow from all
        </Directory>

        ErrorLog /var/log/apache2/error.log
        LogLevel warn

        CustomLog /var/log/apache2/access.log combined
    </VirtualHost>

Now create a file called .gitignore, containing the following. This
prevents the compiled python code being included in the repository and
the archive we use for deployment.

    *.pyc

You should now be ready to initialise a git repository in the top
level project_name directory.

    git init
    git add .gitignore project_name
    git commit -m "Initial commit"

All of that should leave you with 
    
    project_name
        .git
        .gitignore
        requirements.txt
        fabfile.py
        project_name
            __init__.py
            project_name
            project_name.wsgi
            manage.py
            settings.py
            urls.py

In reality you might prefer to keep your wsgi files and virtual host files
elsewhere. The fabfile has a variable (config.virtualhost_path) for this case. 
You'll also want to set the hosts that you intend to deploy to (config.hosts)
as well as the user (config.user).

The first task we're interested in is called setup. It installs all the 
required software on the remote machine, then deploys your code and restarts
the webserver.

    fab local setup

After you've made a few changes and commit them to the master Git branch you 
can run to deply the changes.
    
    fab local deploy

If something is wrong then you can rollback to the previous version.

    fab local rollback
    
Note that this only allows you to rollback to the release immediately before
the latest one. If you want to pick a arbitrary release then you can use the
following, where 20090727170527 is a timestamp for an existing release.

    fab local deploy_version:20090727170527

If you want to ensure your tests run before you make a deployment then you can 
do the following.

    fab local test deploy

"""

from fabric.api import *
import os
# globals
env.project_name = 'project_name' # no spaces!
env.use_daemontools = False  # not available for hardy heron!
env.webserver = 'nginx' # nginx or apache2 (directory name below /etc!)
env.dbserver = 'mysql' # mysql or postgresql
# TODO: database and SSH setup

env.port = 8080 # local FCGI server


# environments

def localhost():
    "Use the local virtual server"
    env.hosts = ['localhost']
    env.user = 'hraban' # You must create and sudo-enable the user first!
    env.path = '/Users/%(user)s/workspace/%(project_name)s' % env # User home on OSX, TODO: check local OS
    env.virtualhost_path = env.path
    env.pysp = '%(virtualhost_path)s/lib/python2.6/site-packages' % env
    env.envname = "test"
    env.init_filename = "localhost.ini"
    env.virtual_env_dest = "."

def alpha_webserver():
    "The alpha (or first) site environment"
    env.hosts = ['webserver.example.com'] # Change to your server name!
    env.user = env.project_name
    env.path = '/var/www/%(project_name)s' % env
    env.virtualhost_path = env.path
    env.pysp = '%(virtualhost_path)s/lib/python2.5/site-packages' % env
    env.envname = "alpha"
    
    env.apache_group = "_www"
    env.apache_user = "_www"


def beta_webserver():
    "The beta (or second) site environment"
    env.hosts = ['webserver.example.com'] # Change to your server name!
    env.user = env.project_name
    env.path = '/var/www/%(project_name)s' % env
    env.virtualhost_path = env.path
    env.pysp = '%(virtualhost_path)s/lib/python2.5/site-packages' % env
    env.envname = "alpha"
    env.config_filename = "development.ini"
    env.virtual_env_dest = "."
    
    env.apache_group = "_www"
    env.apache_user = "_www"
    env.root_user = "root"
    env.root_group = "admin"

def production_webserver():
    "Use the actual webserver"
    env.hosts = ['webserver.example.com'] # Change to your server name!
    env.user = env.project_name
    env.path = '/var/www/%(project_name)s' % env
    env.virtualhost_path = env.path
    env.pysp = '%(virtualhost_path)s/lib/python2.5/site-packages' % env
    env.envname = "production"
    env.config_filename = "production.ini"
    env.virtual_env_dest = "."

    env.apache_group = "_www"
    env.apache_user = "_www"
    env.root_user = "root"
    env.root_group = "admin"


# tasks

def test():
    "Run the test suite and bail out if it fails"
    local("cd %(path)s; python manage.py test" % env) #, fail="abort")
    

def setup():
    """
    Setup a fresh virtualenv as well as a few useful directories, then run
    a full deployment
    """
    require('hosts', provided_by=[localhost,alpha_webserver, beta_webserver, production_webserver])
    require('path')
    
    #sudo('aptitude install -y python-setuptools')
    sudo('easy_install pip')
    sudo('easy_install virtualenv')
    #sudo('aptitude install -y apache2')
    #sudo('aptitude install -y libapache2-mod-wsgi')
    
    sudo('easy_install virtualenv')
    
    # we want rid of the default apache config
    #sudo('cd /etc/apache2/sites-available/; a2dissite default;')
    run('mkdir -p %(path)s; cd %(path)s; virtualenv .;' % env)
    run('cd %(path)s; mkdir releases; mkdir shared; mkdir packages; mkdir data; mkdir python-eggs; chown %(apache_group)s:%(apache_user)s python-eggs' % env)
    deploy()

def deploy():
    """
    Deploy the latest version of the site to the servers, install any
    required third party modules, install the virtual host and 
    then restart the webserver
    """
    require('hosts', provided_by=[localhost,alpha_webserver, beta_webserver, production_webserver])
    require('path')

    import time
    env.release = time.strftime('%Y%m%d%H%M%S')
    upload_tar_from_hg()
    install_requirements()
    install_site()
    symlink_current_release()
    migrate()
    restart_webserver()

def deploy_version(version):
    "Specify a specific version to be made live"
    require('hosts', provided_by=[localhost,alpha_webserver, beta_webserver, production_webserver])
    require('path')
    env.version = version
    with cd(env.path):
        run('rm -rf releases/previous; mv releases/current releases/previous;', pty=True)
        run('ln -s %(version)s releases/current' % env, pty=True)
    restart_webserver()

def rollback():
    """
    Limited rollback capability. Simply loads the previously current
    version of the code. Rolling back again will swap between the two.
    """
    require('hosts', provided_by=[localhost,alpha_webserver, production_webserver])
    require('path')
    with cd(env.path):
        run('mv releases/current releases/_previous;', pty=True)
        run('mv releases/previous releases/current;', pty=True)
        run('mv releases/_previous releases/previous;', pty=True)
    restart_webserver()
    
# Helpers. These are called by other functions rather than directly

def upload_tar_from_hg():
    """"Create an archive from the most current branch that matches the envname
    For example, if you are deploying to the test environment, will look for
    the most recent hg branch that begins with test.
    
    This allows us to be able to tag branches for multiple test environments,
    and deploy them separately. Imagine you have a "alpha" environment, which
    is updated daily, and a "beta" environment, which is only deployed to
    when devs give the OK. This approach with tags allows you to deploy to
    alpha, or beta, even if you're a few commits beyond that point.
    
    Of course, no spaces are allowed in the tag name
    """
    
    require('release', provided_by=[deploy, setup])
    
    whattag = local( 'hg tags | grep ^%(envname)s -m 1 | cut -d " " -f 1' % env ).strip()
    tagname_release = dict(tagname=whattag, release=env.release)
    local( "hg archive -r %(tagname)s -t tar %(release)s.tar.gz" % tagname_release )
    
    # put this as a tag in our repository, so we can match later
    local("hg tag -r %(tagname)s delploy_%(release)s" % tagname_release)
    
    run('mkdir %(path)s/releases/%(release)s' % env)
    put('%(release)s.tar.gz' % env, '%(path)s/packages/' % env)
    
    local_deployment_archives_path = local("pwd").strip() + "/deployment_archives/"
    if not(  os.path.exists(local_deployment_archives_path)  ):
        local("mkdir " + local_deployment_archives_path)
    local( 'mv %(release)s.tar.gz %(ldarcp)s/%(envname)s_%(release)s.tar.gz' % dict(release=env["release"],
        ldarcp=local_deployment_archives_path, envname=env["envname"]) )
    run('cd %(path)s/releases/ && tar zxf ../packages/%(release)s.tar.gz' % env)
    #local('rm %(release)s.tar.gz' % env)

def install_site():
    "Add the virtualhost file to apache"
    require('release', provided_by=[deploy, setup])
    with cd('%(path)s/releases/%(release)s/%(webserver)s_%(envname)s/' % env):
        sudo('cp %(project_name)s.conf /etc/%(webserver)s/sites-available/%(project_name)s.conf' % env, pty=True)
    #sudo('cd /etc/apache2/sites-available/; a2ensite $(project_name)') 

    # the .wsgi file says that the permissions should be so...
    sudo('chown -R %(apache_group)s:%(apache_user)s %(path)s/releases/%(release)s/' % env)
    sudo('chown -R %(root_user)s:%(root_group)s %(path)s/%(virtual_env_dest)s/' % env)
    sudo('chmod 777 %(path)s/releases/%(release)s/%(project_name)s/logs/' % env)
    sudo("ln -s %(path)s/data %(path)s/releases/%(release)s/%(project_name)s/data" % env)
    
    # Build the egg for the site...
    sudo("cd %(path)s/releases/%(release)s/; python setup.py build")

def install_requirements():
    "Install the required packages from the requirements file using pip"
    require('release', provided_by=[deploy, setup])
    run('cd %(path)s; pip install -E %(virtual_env_dest)s -r ./releases/%(release)s/requirements.txt' % env, pty=True)

def symlink_current_release():
    "Symlink our current release"
    require('release', provided_by=[deploy, setup])
    with cd(env.path):
        run('test -e releases/previous && rm releases/previous || echo "no previous deploy found"')
        run('test -e releases/current && mv releases/current releases/previous || echo "no current deploy found"')
        run('ln -s %(release)s releases/current' % env, pty=True)

def migrate():
    "Update the database"
    require( 'release', provided_by=[deploy, setup] )
    
    run ( 'cd %(path)s/releases/current; paster setup-app %(config_filename)s' % env )
    #run('cd $(path)/releases/current/$(project_name);  ../../../bin/python manage.py syncdb --noinput')

def restart_webserver():
    "Restart the web server"
    sudo('/etc/init.d/apache2 restart')