inprogress django to pyramid porting thingy

gistfile1.rst

Porting an Existing Application to Pyramid

Porting an application across platforms is never particularly easy, nor very interesting. In fact, it's usually a terrible idea: in general, if it ain't broke, don't fix it. But sometimes you have to do it. Maybe a new platform has a feature you believe you can't live without, or the project scope is expanding in directions that make you feel like your original platform choice might not have been the best, or maybe it's just necessary to port for political reasons. Whatever.

In this article, I'll describe a way to port an existing Django application to Pyramid. But I won't try to invent a motivation for doing that. Instead, I'm going pretend it has already been decided, and we don't have any choice in the matter: in this scenario, the Flying Spaghetti Monster Himself has told me that I must do this, whether I like it or not. If that's inconceivable to you, please try direct your outrage to Him, not to me. On the other hand, if you find yourself in this boat eventually, and you find this article useful, great. It might also be interesting to existing Django developers to see how to take advantage of Django tools within other environments. I'll gain a better understanding of Django in the process, which is a win to me personally.

Starting Out

The project we're going to port is the djangopeople project. This is the code that runs https://people.djangoproject.com/ . The original code is at https://github.com/simonw/djangopeople.net but we'll be using a more up to date fork of the codebase from https://github.com/brutasse/djangopeople because it runs under more recent Django versions. I've actually forked that fork of the code to my own Github account, in case I need to make bug fixes, and I'll be running from the fork-of-a-fork codebase at https://github.com/mcdonc/djangopeople

The first thing I'll do is get the Django project itself running on my local system. I already have postgres development packages installed so this shouldn't be too hard.

$ cd ~/projects
$ mkdir djangopeople
$ cd djangopeople
$ git clone git@github.com:mcdonc/djangopeople.git
$ virtualenv2.7 env27
$ cd djangopeople
$ ../env27/bin/pip install -r requirements.txt

That failed on my Ubuntu system with this error:

gevent/libevent.h:9:19: fatal error: event.h: No such file or directory

Fix:

sudo apt-get install libevent-dev

Create the database:

$ sudo su - postgres $ createuser Enter name of role to add: chrism Shall the new role be a superuser? (y/n) y $ createdb djangopeople $ psql psql (9.1.5) Type "help" for help.

postgres=# alter user chrism with encrypted password 'chrism'; ALTER ROLE postgres=# grant all privileges on database djangopeople to chrism; GRANT postgres=#

Add the following settings.py to the djangopeople package:

from default_settings import *

DEBUG = True
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'djangopeople',                      
        'USER': 'chrism',
        'PASSWORD': 'chrism',
        'HOST': 'localhost'
    }
}

Activate the virtualenv and populate the database:

$ source ../env27/bin/activate
(env27)[chrism@thinko djangopeople]$ PYTHONPATH=. make db
django-admin.py syncdb --settings=djangopeople.settings --noinput && django-admin.py fix_counts --settings=djangopeople.settings
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Creating table tagging_tag
Creating table tagging_taggeditem
Creating table django_openidconsumer_nonce
Creating table django_openidconsumer_newnonce
Creating table django_openidconsumer_association
Creating table django_openidauth_useropenid
Creating table djangopeople_country
Creating table djangopeople_region
Creating table djangopeople_djangoperson
Creating table djangopeople_portfoliosite
Creating table djangopeople_countrysite
Creating table machinetags_machinetaggeditem
Installing custom SQL ...
Installing indexes ...
Installed 286 object(s) from 1 fixture(s)

Run the server:

(env27)[chrism@thinko djangopeople]$ PYTHONPATH=. make run

It runs. Hooray. Let's copy over a manage.py from another Django project so we can use it to run createsuperuser:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djangopeople.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Use it to create an admin superuser:

(env27)[chrism@thinko djangopeople]$ python manage.py createsuperuser
Username (leave blank to use 'chrism'): admin
E-mail address: chrism@plope.com
Password: 
Password (again): 
Superuser created successfully.

Now we can log in to the Django admin interface successfully. Great. We're done setting things up. We have a working baseline djangopeople install that will run when we type "make run", and we can see it working when we visit http://localhost:8000.

Starting a New Pyramid Project

Our strategy for doing this port is going to be this:

• We're going to set up a new Pyramid project based on the alchemy scaffold. This creates a very barebones Pyramid application which can talk to a database using SQLAlchemy.
• We're going to instruct the Pyramid application to proxy every URL it can't handle to the existing Django application.
• We'll port features incrementally to the Pyramid application, one URL at a time, allowing the old application to continue to respond on its existing URL patterns.

To do this, we'll first install Pyramid into the same virtualenv and create a new Pyramid project. We'll call the Pyramid project pyramidpeople, because, well.. what else would we call it?

Within the directory that contains env27 and the topmost djangopeople directory:

$ env27/bin/pip install pyramid

As of this writing the most recent release of Pyramid is 1.4a1, so that's what gets installed when we do that.

We'll also install pyramid_jinja2 to be able to use Jinja 2 as the templating language, as it's the closest analogue of the Django templating language, so using it will make porting slightly easier than using a different templating language:

# env27/bin/pip install pyramid_jinja2

After we have Pyramid and pyramid_jinja2 installed, we'll use pcreate to create an alchemy Pyramid project:

$ bin/pcreate -s alchemy pyramidpeople

We'll also make a convenience symlink to the djangopeople project within our virtualenv's site-packages directory so we don't have to keep specifying the PYTHONPATH on the command line to point to it:

$ cd env27/lib/python2.7/site-packages/
$ ln -s ../../../../djangopeople/djangopeople .

We'll then install the pyramidpeople project into the virtualenv:

$ cd ~/projects/djangpeople/pyramidpeople
$ ../env27/bin/python setup.py develop

We'll change the sqlalchemy.url in the development.ini file to point to our djangopeople database instead of the default sqlite database:

sqlalchemy.url = postgres://chrism:chrism@localhost/djangopeople

Then we'll install sqlautocode into our virtualenv too. This thing lets us sort of reverse engineer SQLAlchemy model classes from existing database tables:

$ ../env27/bin/pip install sqlautocode

And we'll use sqlautocode to generate a models.py for us:

$ ../env27/bin/sqlautocode -d -o models.py postgres://chrism:chrism@localhost/djangopeople

We'll copy the code from our generated models.py file to our pyramidpeople package's model.py:

$ cat models.py >> pyramidpeople/pyramidpeople/models.py

We have to do some minor surgery to the models.py file we appended to, removing the MyModel class and the Base class created by Pyramid. We'll also remove the engine created by sqlautocode and any assignments to the engine.

We'll edit our pyramidpeople views.py and scripts/initializedb.py to stop trying to import MyModel, instead, for now, importing DjangopeopleDjangoperson instead, just so things import correctly. We'll remove the my_view view within views.py too, because it's useless to us.

We'll also change the static view registration in the Pyramid application from responding on /static to one that responds on /ppstatic:

From:

config.add_static_view('static', 'static', cache_max_age=3600)

To:

config.add_static_view('ppstatic', 'static', cache_max_age=3600)

We do this in order to not override the existing Django /static URLs.

And we'll add an include line to the __init__ that includes the Jinja2 library:

config.include('pyramid_jinja2')

At this point we can start the application, but no views will answer, so we'll be presented with an error when we visit /.

We'll add a new file within the pyramidpeople package:

$ cd ~/projects/djangopeople/pyramidpeople/pyramidpeople
$ emacs proxy.py

The contents of proxy.py will look like this:

import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'djangopeople.settings'

import django.conf
import django.core.handlers.wsgi

class LegacyView(object):
    def __init__(self, app):
        self.app = app
    def __call__(self, request):
        return request.get_response(self.app)

django_view = LegacyView(django.core.handlers.wsgi.WSGIHandler())

Then we'll edit the __init__.py of pyramidpeople and add a notfound view that points to the django_view view:

from pyramid.config import Configurator
from sqlalchemy import engine_from_config

from .models import DBSession
from .proxy import django_view

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    engine = engine_from_config(settings, 'sqlalchemy.')
    DBSession.configure(bind=engine)
    config = Configurator(settings=settings)
    config.add_static_view('static', 'static', cache_max_age=3600)
    config.add_route('home', '/')
    config.add_notfound_view(django_view)
    config.scan()
    return config.make_wsgi_app()

Now if we start the Pyramid app via ../env27/bin/pserve development.ini and visit http://localhost:6543/, we'll see the original Django application served via Pyramid's notfound view. We can tell it's being served by Pyramid at this point because the Pyramid debug toolbar overlays the Djangopeople home page.

It's served when we visit / because there are no Pyramid views registered to answer on / and, thus, the Pyramid notfound view kicks in and the Pyramid notfound view is actually a view which actually serves up the Django aplication. We can also visit /admin and see the Django admin interface. The end result is this: whenever we add a Pyramid view, it will override any existing Django view, but if Pyramid can't find a view, the request will kicked over to the Django application. This means we can begin to change things incrementally one bit at a time, while, at each step still having a working system.

Note that one interesting side effect of how we're doing this is that we can have the original Djangopeople application served under the Django development server (via make run) while it's also being failed back to by Pyramid (via pserve development.ini). We can visit http://localhost:8000 to see the original application, and we can visit http://localhost:6543/ to see the Pyramid-proxied version of the same application. This is useful for comparing before and after behavior without much effort.

We'll now comment out the Pyramid debug toolbar from development.ini. Instead of:

pyramid.includes =
    pyramid_debugtoolbar
    pyramid_tm
    pyramid_jinja2

We'll have:

pyramid.includes =
#    pyramid_debugtoolbar
    pyramid_tm
    pyramid_jinja2

This is because the toolbar doesn't seem to want to play nicely with proxy form posts at the moment. This requires a restart of the Pyramid application to take effect.

I applied a patch to djangopeople fork to expose the privacy_im select field in the signup form to fix a small bug that prevented signups from working: https://github.com/mcdonc/djangopeople/commit/34d6ef504c1c8d4fd20d2532e80a6ec4af864341

Porting: The First Step

The first things we're going to do will be to:

• Copy the djangopeople static files into the pyramidpeople project.
• Convert the djangopeople base.html template into a Jinja2 template within the pyramidpeople's templates directory.

Copying Static Files

We'll remove any existing files from the Pyramid application's static dir (they're unused):

$ rm ~/projects/djangopeople/pyramidpeople/pyramidpeople/static/*

We'll copy the static files from the djangopeople project into the pryamidpeople project:

$ pwd
/home/chrism/projects/djangopeople/djangopeople/djangopeople/djangopeople/static/djangopeople
$ cp -r . ~/projects/djangopeople/pyramidpeople/pyramidpeople/static/

This leaves us with css, img, and js directories within pyramdidpeople/static. We've already got a view registered for /ppstatic in the Pyramid application which serves us files from there. If we visit http://localhost:6543/ppstatic/img/bullet_green.png we can see that these files are now being served.

Adding A Session Factory

This django application makes use of sessions. To support sessions, we'll add a "session factory" to our __init__.py and add a session.key setting to our development.ini file. Here's what we add to __init__.py:

from pyramid.session import UnencryptedCookieSessionFactoryConfig
sessionfactory = UnencryptedCookieSessionFactoryConfig(settings['session.secret'])
config.set_session_factory(sessionfactory)

And here's what we add to development.ini:

session.secret = sosecret

After this change, the session object will be available as request.session.

Porting the Django Template to Jinja2

We'll copy the base.html template from within the djangopeople project to a file named base.jinja2 within the templates subdirectory of the pyramidpeople project and we'll hack the everloving crap out of it.

We have to replace the use of the {% url %} tag with calls to request.route_url(). So {% url "fred" %} becomes {{ request.route_url('fred') }}. We actually alias request.route_url to route_url at the top of the template by doing {% set route_url = request.route_url %}} so we can do {{ route_url('fred') }} instead of the longer variant. We do a similar thing for the {% static %} tag, replacing e.g. {% static 'djangopeople/some/file/name.css' %} with something like {{ static_url('pyramidpeople:static/some/file/name.css') }}.

Note that we're making use of the Pyramid static_url method here. Pyramid's static view machinery has no notion of an ordered static file search path, nor does it require you to put all of your static files in the same directory. Instead, the string we pass to static_url is an asset specification. The asset specification pyramidpeople:static/some/file/name.css means "the file relative to the root of the pyramidpeople Python package named static/some/file/name.css. On my hard drive that file would be addressable via /home/chrism/projects/djangopeople/pyramidpeople/pyramidpeople/static/some/file/name.css, but Pyramid doesn't make you use filenames. It wants a package-relative asset spec, because a Python package might move around on disk. The URL generated by static_url('pyramidpeople:static/some/file/name.css') is http://localhost:6543/ppstatic/some/file/name.css. Pyramid's static view machinery generates this url and serves up the file as necessary, based on the add_static_view line in the __init__.py. So files can live wherever you want, as long as they live in a Python package, and as long as you've created a add_static_view statement which allows them to be served up.

We're also using the Pyramid request.route_url API. This is just a function that accepts a route name and returns a URL for that route. It is capable of accepting a bunch of arguments, but we don't really care about that yet, because we're mostly generating nondynamic, static URLs.

We left some things undone and worked around some things.

Jinja2 has no notion of the {% blocktrans %} tag, or at least I'm too lazy to look if it does or not, so we (at least for now) stop using it. It also doesn't accept the syntax {% trans 'foo' %}. We have to convert that syntax to {% trans %}foo{% endtrans %}.

Our template port is pretty mechanical. We could have chosen to do less typing and more coding, introducing url and static extensions to Jinja2 instead of replacing the {% url %} and {% static %} tags, but I chose not to.

Adding a Test View

We'll add a view to the pyramidpeople views.py to attempt to render the main template by itself:

from pyramid.view import view_config

class DummyUser(object):
    username = 'fred'

@view_config(
    route_name='test', 
    renderer='pyramidpeople:templates/base.jinja2'
    )
def test(request):
    user = DummyUser()
    return {'user':user}

We supply the template with a value named user because we know it wants one in order to render.

In order to hook the view up to a URL, we'll add a route named test within our __init__.py:

config.add_route('test', '/test')

When we restart the Pyramid app and try to render this view by visiting /test, we encounter an error: KeyError: 'No such route named index'. This is because the template calls route_url('index') when it tries to render itself, and there's no route named index. We'll fix that by adding one in __init__.py:

config.add_route('index', '/')

Once we do that, and try to re-render the app we wind up with a similar error for about. We need to add all of the routes it wants.:

config.add_route('index', '/')
config.add_route('about', '/about/')
config.add_route('search', '/search/')
config.add_route('login', '/login/')
config.add_route('signup', '/signup/')
config.add_route('redirect_to_logged_in_user_profile',
                 '/redirect_to_logged_in_user_profile/')
config.add_route('openid_begin', '/openid_begin/')

At this point, upon a restart, the page renders. It looks uglier than sin, because it has no styling, but it renders:

All of the page's regular links are functional, and send us to the correct place. The login link drops down a form div, but when the form is posted, it leads to an error because the CRSF token is incorrect from Django's perspective. And there's no link to the redirect route. But we can see if we view source that all the head links are pointing to functional links and that we've actually generated a CSRF token in the body.

Note that even though we've added Pyramid routes at this point, we don't have any views hooked up to them, so they're basically inert. We can generate route urls using the routes we've added via route_url, but when we visit any of the URL patterns mentioned in the routes, Pyramid still raises a notfound error, which takes us over to Djangoland. This is exactly what we want right now.

One thing we've figured out from just doing this little bit of work is that if we want to be able to replace the Django application with its Pyramid counterpart bit-by-bit we're going to need the Pyramid application to share authentication, session, and CSRF data with the Django application. It's boring to invent this compatibility layer, so I think I'm going to bail temporarily on the original idea of overlaying the Django application with the Pyramid one and porting incrementally, with the entire URL-space representing a working application. It can be done, but I'd rather not get bogged down in the details of crossplatform cookie and session compatibility right now, because I'm writing this as I'm going, and writing as I do that would make for some even duller reading than this already is. So I'm going to continue by attempting to implement the view and template that backs the /login/ URL.

Implementing the Login URL

We have to do a little package restructuring and software installation before we move on.

First of all, we're going to install the WTForms package, which is a package that works a lot like the Django forms system.:

$ ~/projects/djangopeople/env27/bin/pip install WTForms

There are plenty of other form systems. We'll be using this one to stay as close to the way things work in the original Django application as possible.

Then we're going to turn the views.py module in our pyramidproject project into a package. I like to create subpackages in my project that are self-contained. So instead of placing all views into a top-level views module, all models in a top-level models module, and all templates into the same templates directory, I like to create subpackages along functional lines and then create only the views, models, and templates that relate to that functionality within them. So I'm going to create an auth subpackage in our pyramidpeople package, and a templates directory underneath that which we'll use to put our login-related templates in:

$ cd ~/projects/djangopeople/pyramidpeople/pyramidpeople
$ mkdir -p auth/templates
$ touch auth/__init__.py

I'm then going to start fleshing out the auth module by copying the login.html template from the djangopeople templates directory into the templates subdirectory of our new package.:

$ cd ~/projects/djangopeople/pyramidpeople/pyramidpeople/auth/templates
$ cp ~/projects/djangopeople/djangopeople/djangopeople/djangopeople/templates/login.html login.jinja2

• Need to use asset spec in extends.
• Same old stuff in template otherwise.
• Stopped scanning whole package, created includeme in auth which scans it locally. Moved an add_route to that includeme too for the login view.
• Had = missing between "link rel" and "stylesheet" that was preventing styles.css from being loaded.