Skip to content

Instantly share code, notes, and snippets.

@kgarnett29

kgarnett29/gist:edf65f0028896bf552b82852472ecc94

Forked from evildmp/gist:3094281
Created March 12, 2018 15:07
Download ZIP

Revisions

  1. @evildmp evildmp revised this gist Oct 7, 2013. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,8 @@
    This document has now been incorporated into the uWSGI documentation::

    http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html


    ******************************
    Set up Django, nginx and uwsgi
    ******************************
  2. @superdmp superdmp revised this gist Jul 16, 2012. 1 changed file with 223 additions and 488 deletions.
    711 changes: 223 additions & 488 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -1,630 +1,365 @@
    ##############################################
    Setting up an Arkestra web server from scratch
    ##############################################
    ******************************
    Set up Django, nginx and uwsgi
    ******************************

    ************************
    Basic Debian Linux setup
    ************************
    Steps with explanations to set up a server using:

    In this document, most of the lines in code sections can simply be copied and pasted into the terminal.
    * virtualenv
    * Django
    * nginx
    * uwsgi

    Lines that follow each other immediately can generally be pasted in as one block. Lines separated by whitespace may require some user input between them, so should not be pasted as one block::

    # these two lines can be copied
    # and pasted as one block

    # don't paste this one in the same block

    Obtain Debian Linux
    ===================

    Get a Debian network installer from http://www.debian.org (I used debian-6.0.5-amd64-netinst.iso).

    Copy it onto a USB flash drive::

    diskutil unmountDisk /dev/disk1 # unmount the volume
    cat Desktop/debian-6.0.5-amd64-netinst.iso | dd of=/dev/disk1 :# copy the disk image

    Boot up the machine from the flash drive, and let Debian use its suggested defaults.

    When it comes to ask what software to install, tell it to install only:

    * basic system utilities
    * ssh server

    Configure the system
    ====================

    You'll need to login via ssh then su to root::

    su root

    Now you can login as from another machine instead of from the console. So do that now.

    As root::

    apt-get install sudo # now you can use sudo

    adduser arkestra sudo # add your user account to the sudoers file

    We don't need to be root anymore, so log out of root. You can't use sudo until you log in again though.

    Login as arkestra::

    sudo locale-gen en_GB.UTF-8 # set up locales
    sudo /usr/sbin/update-locale LANG=en_GB.UTF-8

    sudo apt-get update # shouldn't really be necessary, as this is a new install
    sudo aptitude safe-upgrade

    # installing handy things for compiling and building
    sudo aptitude install build-essential linux-headers-`uname -r`

    Make your life easier: having /sbin on the path provides access to commands otherwise reserved for root::

    pico ~/.profile # edit the user's profile

    add::

    PATH=$PATH:/sbin
    export PATH

    Log out then in again.

    ****************
    ssh and firewall
    ****************

    OpenSSH
    Concept
    =======

    ::

    ssh-keygen # generate server keys # accept defaults & don't create a passphrase

    Get your own public key(s) into and paste them into .ssh/authorized_keys::

    # get your local key - not on the server, on your local machine!
    less ~/.ssh/id_rsa.pub

    On the server::

    # edit the authorized_keys file on the server and paste them in
    pico ~/.ssh/authorized_keys
    nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.

    # make the permissions stricter
    chmod 600 ~/.ssh/authorized_keys
    That's uwsgi's job. uwsgi will create a Unix socket, and serve responses to nginx via the uwsgi protocol - the socket passes data in both directions::

    Edit ssh_config::
    the outside world <-> nginx <-> the socket <-> uwsgi

    sudo pico /etc/ssh/sshd_config
    Before you start
    ================

    Change the following 3 options::
    virtualenv
    ----------

    Port 22
    PermitRootLogin yes
    PasswordAuthentication yes
    Make sure you are in a virtualenv - you will install a system-wide uwsgi later.

    to::
    Django
    ------

    Port 8888
    PermitRootLogin no
    PasswordAuthentication no
    I am assuming Django 1.4. It automatically creates a wsgi module when you create a project. If you're using an earlier version, you will have to find a Django wsgi module for your project.

    Note that I'm also assuming a Django 1.4 project structure, in which you see paths like::

    The firewall
    ============
    /path/to/your/project/project/

    ::

    # install the Unix firewall
    sudo apt-get install ufw

    #open some ports
    sudo ufw allow 8000/tcp # web dev
    sudo ufw allow 8001/tcp # web dev
    sudo ufw allow 8002/tcp # web dev
    sudo ufw allow 8003/tcp # web dev
    sudo ufw allow 8004/tcp # web dev
    sudo ufw allow 8005/tcp # web dev
    sudo ufw allow 8082/tcp # munin
    sudo ufw allow 8888/tcp # for ssh
    sudo ufw allow 80/tcp # standard http
    sudo ufw allow 22/tcp # ssh
    # enable the firewall
    sudo ufw enable

    # restart sshd
    # make sure you do this as root, using sudo
    sudo /etc/init.d/ssh restart

    From now on you'll need to ssh on on port 8888.

    ************
    MySQL server
    ************

    ::

    # install mysql server and python interface
    sudo apt-get install mysql-server python-mysqldb

    Set a root password for MySQL.

    ::

    sudo pico /etc/mysql/my.cnf # edit system-wide MySQL conf

    Add (or if necessary change) in the [mysqld] section::

    skip-character-set-client-handshake
    collation_server = utf8_unicode_ci
    character_set_server = utf8

    ::

    sudo /etc/init.d/mysql restart

    # test the server
    mysql -u root -p # login

    create database bongo; # create a database
    show create database bongo; # see what's been created

    You need to see that it's using the correct character set:
    (i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.

    `CREATE DATABASE 'bongo' /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci *`
    It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.

    ::

    # drop the database
    drop database bongo;


    *****
    nginx
    *****

    The version of Nginx from Debian stable is rather old. We'll install from backports::

    sudo pico /etc/apt/sources.list # edit the sources list

    Add::

    # backports
    deb http://backports.debian.org/debian-backports squeeze-backports main
    About the domain and port
    -------------------------

    You must update aptitude's records::
    I'll call your domain example.com. Substitute your own FQDN or IP address.

    sudo apt-get update
    sudo apt-get -t squeeze-backports install nginx # install nginx
    Throughout, I'm using port 8000. You can use whatever port you want of course, but I have chosen this one so it doesn't conflict with anything a web server might be doing already.

    sudo /etc/init.d/nginx start # start nginx
    Basic uwsgi intallation and configuration
    =========================================

    And now check that the server is serving by visiting it in a web browser. You'll get a "Welcome to nginx!" message.

    Add www-data (default nginx user) to the arkestra group, so they can conveniently share folders::

    sudo usermod -a -G arkestra www-data

    **************
    Other software
    **************

    Version control
    ===============
    Install uwsgi
    -------------

    ::

    sudo apt-get install git mercurial # install Git and Mercurial
    pip install uwsgi

    Basic test
    ----------

    Python
    ======
    Create a file called test.py::

    ::

    sudo apt-get install python-virtualenv python-dev # install virtualenv & development version of Python
    # test.py
    def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return "Hello World"

    sudo apt-get install python-ldap # install ldap support
    Run::

    uwsgi --http :8000 --wsgi-file test.py

    Video
    =====
    The options mean:

    We install Handbrake and ffmpeg2theora::
    http :8000
    use protocol http, port 8000

    sudo pico /etc/apt/sources.list # edit the sources list (because handbrake isn't otherwise available)
    wsgi-file test.py
    load the specified file

    Add::
    This should serve a hello world message directly to the browser on port 8000. Visit::

    #Handbrake
    deb http://www.debian-multimedia.org squeeze main # use this multmedia repository
    http://example.com:8000

    Then::
    to check.

    sudo apt-get update
    sudo apt-get install handbrake-cli ffmpeg2theora
    Test your Django project
    ------------------------

    Now we want uwsgi to do the same thing, but to run a Django site instead of the test.py module.

    Miscellaneous libraries
    =======================
    But first, make sure that your project actually works! Now you need to be in your Django project directory.

    ::

    sudo apt-get install libxml2-dev libxslt-dev libjpeg8-dev # install some libraries

    python manage.py runserver 0.0.0.0:8000

    uwsgi
    =====
    Now run it using uwsgi::

    Install uwsgi for the system::
    uwsgi --http :8000 --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv

    sudo pip install uwsgi
    The options mean:

    Django & Arkestra
    =================

    ::
    chdir /path/to/your/project
    use your Django project directory as a base
    module project.wsgi
    i.e. the Python wsgi module in your project
    virtualenv /path/to/virtualenv
    the virtualenv

    cd # just make sure we're in ~/
    virtualenv arkestra # the virtual environment that will contain all our Python software
    cd arkestra
    source bin/activate
    There is an alternative to using the `--module` option, by referring instead to the wsgi *file*::

    pip install -e git+https://github.com/evildmp/Arkestra.git@pip#egg=Arkestra # install Arkestra & dependencies
    wsgi-file /path/to/your/project/project/wsgi.py
    i.e. the system file path to the wsgi.py file

    It's necessary to keep an eye on the output of the installation procedure - sometimes a server hosting some dependency won't be available, so make sure there are no errors. If necessary, just run it again.

    Install the right versions of Semantic Editor, Django Widgetry and Django Filer::
    Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.

    pip install -r src/arkestra/REQUIREMENTS.txt # install the things that pip can't do automatically
    Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.

    Basic nginx
    ===========

    *********************
    Test the example site
    *********************
    Install nginx
    -------------

    You can skip to *Set up a production server* if you don't need to test this.
    The version of Nginx from Debian stable is rather old. We'll install from backports.

    ::

    cd src/arkestra/example_14
    python manage.py syncdb # answer "no"
    sudo pico /etc/apt/sources.list # edit the sources list

    python manage.py reset contenttypes # answer "yes"

    python manage.py loaddata example_database.json
    python manage.py runserver 0.0.0.0:8000
    Add::

    The admin username and password are both "arkestra"

    Check that media and static files are being served correctly by the runserver; visit:
    # backports
    deb http://backports.debian.org/debian-backports squeeze-backports main

    * /media/media.png
    * /static/static.png
    Run::

    sudo apt-get -t squeeze-backports install nginx # install nginx
    sudo /etc/init.d/nginx start # start nginx

    **************************
    Set up a production server
    **************************
    And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"

    Configure nginx for your site
    =============================

    Turn off the default site installed by nginx::

    sudo rm /etc/nginx/sites-enabled/default # we remove the default symlink

    Check that nginx has installed a file at /etc/nginx/uwsgi_params::

    ls /etc/nginx/uwsgi_params

    If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it:

    ::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params


    nginx for the example site
    ==========================

    You can skip to *Set up the School of Medicine site* if you don't need to test this.

    Create a file called nginx.conf in the project directory::

    pico ~/arkestra/src/arkestra/example_14/example_14_nginx.conf

    and put this in it::

    # example_14_nginx.conf
    upstream django {
    # connect to this socket
    server unix:///tmp/arkestra_example.sock; # a file socket
    }

    server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name .example.com; # substitute the machine's IP address or FQDN
    charset utf-8;

    #Max upload size
    client_max_body_size 75M;

    # Django media
    location /media {
    alias /home/arkestra/arkestra/src/arkestra/example_14/example_14/media;
    }

    location /static {
    alias /home/arkestra/arkestra/src/arkestra/example_14/example_14/static;
    }

    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass django;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }
    -----------------------------

    Check that your nginx has installed a file at `/etc/nginx/uwsgi_params`. If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params

    Create a file called nginx.conf, and put this in it::

    # nginx.conf
    upstream django {
    # connect to this socket
    # server unix:///tmp/uwsgi.sock; # for a file socket
    server 127.0.0.1:8001; # for a web port socket
    }
    server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name .example.com; # substitute your machine's IP address or FQDN
    charset utf-8;
    #Max upload size
    client_max_body_size 75M; # adjust to taste

    # Django media
    location /media {
    alias /path/to/your/project/project/media; # your Django project's media files
    }
    location /static {
    alias /path/to/your/project/project/static; # your Django project's static files
    }
    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass django;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }

    Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::

    sudo ln -s ~/arkestra/src/arkestra/example_14/example_14_nginx.conf /etc/nginx/sites-enabled/

    Install static files::
    sudo ln -s ~/path/to/your/project/nginx.conf /etc/nginx/sites-enabled/

    python manage.py collectstatic -l
    Basic nginx test
    ----------------

    Restart nginx::

    sudo /etc/init.d/nginx restart
    sudo /etc/init.d/nginx restart

    Check that media files are being served correctly:

    * /media/media.png

    If this works, you'll know at least that nginx is serving files correctly.


    uwsgi for the example site
    ==========================

    Create a file called django.ini in the project directory::

    pico ~/arkestra/src/arkestra/example_14/example_14.ini

    and put this in it::

    # django.ini file
    [uwsgi]
    Add an image called media.png to the /path/to/your/project/project/media directory

    # master
    master = true
    Visit

    # maximum number of processes
    processes = 10
    http://example.com:8000/media/media.png

    # the socket (use the full path to be safe)
    socket = /tmp/arkestra_example.sock

    # 666 is very permissive, but this is just for testing
    chmod-socket = 666

    # the base directory
    chdir = /home/arkestra/arkestra/src/arkestra/example_14

    # Django's wsgi file
    module = example_14.wsgi

    # the virtualenv
    home = /home/arkestra/arkestra

    # clear environment on exit
    vacuum = true

    And run uswgi using the file::

    uwsgi --ini example_14.ini


    Set up the School of Medicine site
    ==================================

    We will use the existing virtualenv, and put the project in ~/.

    Install django CMS from git (we need to do this because we'll have to merge a couple of patches specific to the School site)::

    # make sure the virtualenv is activated before you do this!
    pip install -e git+https://github.com/divio/django-cms.git@master#egg=django-cms # django CMS 2.3 from github

    cd ~/arkestra/src/django-cms # this is where pip should have put django-cms
    git remote add evildmp https://github.com/evildmp/django-cms.git # add this as a git remote repository
    git fetch evildmp # fetch information about what's available from the remote
    git merge evildmp/menu-speed-up # merge the branch that speeds up the menus for us
    git merge evildmp/pageflags # merge the pageflags branch we need

    pip install -e git+https://github.com/evildmp/arkestra-publications.git#egg=arkestra_publications # publications and symplectic applications

    pip install django_auth_ldap # LDAP module; we need this for admin users

    Ignore this for now - we aren't using celery just yet.
    If this works, you'll know at least that nginx is serving files correctly.

    #pip install -e git+https://github.com/celery/celery.git#egg=celery
    #pip install -e git+https://github.com/celery/django-celery.git#egg=django-celery
    nginx and uwsgi and test.py
    ===========================

    #sudo apt-get install rabbitmq-server
    Let's get nginx to speak to the hello world test.py application.

    ::

    Set up the arkestra_medic database and user
    -------------------------------------------
    uwsgi --socket :8001 --wsgi-file test.py

    You'll need a dump of the live database.
    This is nearly the same as before, except now we are not using http between uwsgi and nginx, but the (much more efficient) uwsgi protocol, and we're doing it on port 8001. nginx meanwhile will pass what it finds on that port to port 8000. Visit:

    Create the database, and import the dumped database file.
    http://example.com:8000/

    Create a MySQL user "<name of database user>" and give it all permissions for the database.
    to check.

    Set up the arkestra_medic project folder
    ----------------------------------------
    Meanwhile, you can try to have a look at the uswgi output at:

    The School of Medicine site (settings, templates, CSS etc) is on GitHub. It's publicly-available, and doesn't include:
    http://example.com:8001/

    * sensitive settings (passwords etc)
    * Filer files (images, videos, etc)
    but quite probably, it won't work because your browser speaks http, not uwsgi.

    ::
    Using sockets instead of ports
    ==============================

    # we'll install this in ~/
    cd
    It's better to use Unix sockets than ports - there's less overhead.

    Clone the arkestra_medic project directory::
    Edit nginx.conf.

    git clone https://github.com/evildmp/arkestra_medic.git
    uncomment
    server unix:///tmp/uwsgi.sock;
    comment out
    server 127.0.0.1:8001;

    The media directory needs to be group writable, so nginx can save things in it::
    and restart nginx.

    sudo chgrp -R www-data ~/arkestra_medic/arkestra_medic/media
    sudo chmod -R g+w ~/arkestra_medic/arkestra_medic/media
    Runs uwsgi again::

    In ~/arkestra_medic/arkestra_medic/settings.py, some settings need to be set or changed - they are marked with the string *change this!* and include:
    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py

    * DATABASES (you created the database and users earlier, so you know what they should be)
    * SECRET_KEY (some long random string)
    Try http://example.com:8000/ in the browser.

    Copy media files (from the current School site)::
    If that doesn't work
    --------------------

    scp -r -P 8888 <user>@<fully-qualified-domain-name-of-site>:</path/to/filer/directory> ~/arkestra_medic/arkestra_medic/media/filer
    Check your nginx error log(/var/log/nginx/error.log). If you see something like::

    connect() to unix:///path/to/your/project/uwsgi.sock failed (13: Permission denied)

    Test it all runs::
    then probably you need to manage the permissions on the socket (especially if you are using a file not in /tmp as suggested).

    python manage.py runserver 0.0.0.0:8001 # 8001 because nginx may already be using 8000, if you tested it with the example project
    Try::

    Create a file called arkestra_medic_nginx.conf in the project directory::
    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=644 # 666 permissions (very permissive)

    pico ~/arkestra_medic/arkestra_medic_nginx.conf
    or::

    and put this in it::
    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=664 # 664 permissions (more sensible)

    # nginx.conf
    upstream arkestra_medic {
    # connect to this socket
    server unix:///tmp/arkestra_medic.sock; # a file socket
    }
    You may also have to add your user to nginx's group (probably www-data), or vice-versa, so that nginx can read and write to your socket properly.

    server {
    # the port your site will be served on
    listen 80;
    # the domain name it will serve for
    server_name .example.com; # substitute the machine's IP address or FQDN
    charset utf-8;
    Running the Django application with uswgi and nginx
    ===================================================

    #Max upload size
    client_max_body_size 75M;
    Let's run our Django application::

    # Django media
    location /media {
    alias /home/arkestra/arkestra_medic/arkestra_medic/media;
    }
    uwsgi --socket /tmp/uwsgi.sock --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv --chmod-socket=664

    location /static {
    alias /home/arkestra/arkestra_medic/arkestra_medic/static;
    }
    Now uwsgi and nginx should be serving up your Django application.

    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass arkestra_medic;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }

    Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::

    sudo ln -s ~/arkestra_medic/arkestra_medic_nginx.conf /etc/nginx/sites-enabled/
    a uwsgi .ini file for our Django application
    ============================================

    Install static files::
    Deactivate your virtualenv::

    cd ~/arkestra_medic
    python manage.py collectstatic -l
    deactivate

    Restart nginx::
    and install uwsgi system-wide::

    sudo /etc/init.d/nginx restart
    sudo pip install uwsgi
    We can put the same options that we used with uwsgi into a file, and then ask uwsgi to run with that file::

    Check that media files are being served correctly:
    # django.ini file
    [uwsgi]

    * /media/media.png
    # master
    master = true

    If this works, you'll know at least that nginx is serving files correctly.
    # maximum number of processes
    processes = 10

    Create a file called arkestra_medic_uwsgi.ini in the project directory::
    # the socket (use the full path to be safe)
    socket = /tmp/uwsgi.sock

    pico ~/arkestra_medic/arkestra_medic_uwsgi.ini
    # with appropriate permissions - *may* be needed
    # chmod-socket = 664

    and put this in it::
    # the base directory
    chdir = /path/to/your/project

    # django.ini file
    [uwsgi]
    # Django's wsgi file
    module = project.wsgi

    # master
    master = true
    # the virtualenv
    home = /path/to/virtualenv

    # maximum number of processes
    processes = 10
    # clear environment on exit
    vacuum = true

    # the socket (use the full path to be safe)
    socket = /tmp/arkestra_medic.sock

    # 666 is very permissive, but you can use this if necessary just for testing
    # chmod-socket = 666

    # the base directory
    chdir = /home/arkestra/arkestra_medic

    # Django's wsgi file
    module = arkestra_medic.wsgi
    And run uswgi using the file::

    # the virtualenv
    home = /home/arkestra/arkestra
    uwsgi --ini django.ini

    # clear environment on exit
    vacuum = true
    Note:

    Deactivate your virtualenv::
    --ini django.ini
    use the specified .ini file

    deactivate
    Test emperor mode
    =================

    And run uswgi using the file::
    uwsgi can run in 'emperor' mode. In this mode it keeps an eye on a directory of uwsgi config files, and spawns instances ('vassals') for each one it finds.

    sudo uwsgi --ini arkestra_medic_uwsgi.ini --uid www-data --gid www-data
    Whenever a config file is amended, the emperor will automatically restart the vassal.

    ::

    Edit /etc/rc.local::
    # symlink from the default config directory to your config file
    sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/

    sudo pico /etc/rc.local
    # run the emperor as root
    sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master

    and add::
    The options mean:

    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
    emperor /etc/uwsgi/vassals
    look there for vassals (config files)
    uid www-data
    run as www-data once we've started
    gid www-data
    run as www-data once we've started

    before the line `"exit 0"`.
    Check the site; it should be running.

    ::
    Make uwsgi startup when the system boots
    ========================================

    # symlink from the default config directory to your config file
    sudo mkdir -p /etc/uwsgi/vassals/
    sudo ln -s ~/arkestra_medic/arkestra_medic_uwsgi.ini /etc/uwsgi/vassals/
    The last step is to make it all happen automatically at system startup time.

    Just for now, because the current live server is old, we need to change a couple of items in the database - adjust contenttypes table and filer_video table
    Edit /etc/rc.local and add::

    * django_content_type id 212, change app_name from filer to video
    * rename filer_video table to video_video
    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master

    Reboot the server and it should work at startup::
    before the line "exit 0".

    sudo reboot
    And that should be it!
  3. @superdmp superdmp revised this gist Jul 16, 2012. 1 changed file with 488 additions and 223 deletions.
    711 changes: 488 additions & 223 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -1,365 +1,630 @@
    ******************************
    Set up Django, nginx and uwsgi
    ******************************
    ##############################################
    Setting up an Arkestra web server from scratch
    ##############################################

    Steps with explanations to set up a server using:
    ************************
    Basic Debian Linux setup
    ************************

    * virtualenv
    * Django
    * nginx
    * uwsgi
    In this document, most of the lines in code sections can simply be copied and pasted into the terminal.

    Concept
    =======
    Lines that follow each other immediately can generally be pasted in as one block. Lines separated by whitespace may require some user input between them, so should not be pasted as one block::

    # these two lines can be copied
    # and pasted as one block

    # don't paste this one in the same block

    Obtain Debian Linux
    ===================

    Get a Debian network installer from http://www.debian.org (I used debian-6.0.5-amd64-netinst.iso).

    Copy it onto a USB flash drive::

    diskutil unmountDisk /dev/disk1 # unmount the volume
    cat Desktop/debian-6.0.5-amd64-netinst.iso | dd of=/dev/disk1 :# copy the disk image

    Boot up the machine from the flash drive, and let Debian use its suggested defaults.

    When it comes to ask what software to install, tell it to install only:

    nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.
    * basic system utilities
    * ssh server

    That's uwsgi's job. uwsgi will create a Unix socket, and serve responses to nginx via the uwsgi protocol - the socket passes data in both directions::
    Configure the system
    ====================

    the outside world <-> nginx <-> the socket <-> uwsgi
    You'll need to login via ssh then su to root::

    Before you start
    ================
    su root

    virtualenv
    ----------
    Now you can login as from another machine instead of from the console. So do that now.

    Make sure you are in a virtualenv - you will install a system-wide uwsgi later.
    As root::

    Django
    ------
    apt-get install sudo # now you can use sudo

    I am assuming Django 1.4. It automatically creates a wsgi module when you create a project. If you're using an earlier version, you will have to find a Django wsgi module for your project.
    adduser arkestra sudo # add your user account to the sudoers file

    Note that I'm also assuming a Django 1.4 project structure, in which you see paths like::
    We don't need to be root anymore, so log out of root. You can't use sudo until you log in again though.

    /path/to/your/project/project/
    Login as arkestra::

    (i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.
    sudo locale-gen en_GB.UTF-8 # set up locales
    sudo /usr/sbin/update-locale LANG=en_GB.UTF-8

    It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.
    sudo apt-get update # shouldn't really be necessary, as this is a new install
    sudo aptitude safe-upgrade

    About the domain and port
    -------------------------
    # installing handy things for compiling and building
    sudo aptitude install build-essential linux-headers-`uname -r`

    I'll call your domain example.com. Substitute your own FQDN or IP address.
    Make your life easier: having /sbin on the path provides access to commands otherwise reserved for root::

    Throughout, I'm using port 8000. You can use whatever port you want of course, but I have chosen this one so it doesn't conflict with anything a web server might be doing already.
    pico ~/.profile # edit the user's profile

    Basic uwsgi intallation and configuration
    =========================================
    add::

    Install uwsgi
    -------------
    PATH=$PATH:/sbin
    export PATH

    Log out then in again.

    ****************
    ssh and firewall
    ****************

    OpenSSH
    =======

    ::

    pip install uwsgi
    ssh-keygen # generate server keys # accept defaults & don't create a passphrase

    Get your own public key(s) into and paste them into .ssh/authorized_keys::

    # get your local key - not on the server, on your local machine!
    less ~/.ssh/id_rsa.pub

    On the server::

    # edit the authorized_keys file on the server and paste them in
    pico ~/.ssh/authorized_keys

    Basic test
    ----------
    # make the permissions stricter
    chmod 600 ~/.ssh/authorized_keys

    Create a file called test.py::
    Edit ssh_config::

    # test.py
    def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return "Hello World"
    sudo pico /etc/ssh/sshd_config

    Run::
    Change the following 3 options::

    uwsgi --http :8000 --wsgi-file test.py
    Port 22
    PermitRootLogin yes
    PasswordAuthentication yes

    The options mean:
    to::

    http :8000
    use protocol http, port 8000
    Port 8888
    PermitRootLogin no
    PasswordAuthentication no

    wsgi-file test.py
    load the specified file

    This should serve a hello world message directly to the browser on port 8000. Visit::
    The firewall
    ============

    http://example.com:8000
    ::

    to check.
    # install the Unix firewall
    sudo apt-get install ufw

    #open some ports
    sudo ufw allow 8000/tcp # web dev
    sudo ufw allow 8001/tcp # web dev
    sudo ufw allow 8002/tcp # web dev
    sudo ufw allow 8003/tcp # web dev
    sudo ufw allow 8004/tcp # web dev
    sudo ufw allow 8005/tcp # web dev
    sudo ufw allow 8082/tcp # munin
    sudo ufw allow 8888/tcp # for ssh
    sudo ufw allow 80/tcp # standard http
    sudo ufw allow 22/tcp # ssh
    # enable the firewall
    sudo ufw enable

    # restart sshd
    # make sure you do this as root, using sudo
    sudo /etc/init.d/ssh restart

    From now on you'll need to ssh on on port 8888.

    ************
    MySQL server
    ************

    Test your Django project
    ------------------------
    ::

    Now we want uwsgi to do the same thing, but to run a Django site instead of the test.py module.
    # install mysql server and python interface
    sudo apt-get install mysql-server python-mysqldb

    But first, make sure that your project actually works! Now you need to be in your Django project directory.
    Set a root password for MySQL.

    ::

    python manage.py runserver 0.0.0.0:8000
    sudo pico /etc/mysql/my.cnf # edit system-wide MySQL conf

    Add (or if necessary change) in the [mysqld] section::

    skip-character-set-client-handshake
    collation_server = utf8_unicode_ci
    character_set_server = utf8

    ::

    sudo /etc/init.d/mysql restart

    # test the server
    mysql -u root -p # login

    create database bongo; # create a database
    show create database bongo; # see what's been created

    You need to see that it's using the correct character set:

    `CREATE DATABASE 'bongo' /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci *`

    ::

    # drop the database
    drop database bongo;


    *****
    nginx
    *****

    The version of Nginx from Debian stable is rather old. We'll install from backports::

    Now run it using uwsgi::
    sudo pico /etc/apt/sources.list # edit the sources list

    uwsgi --http :8000 --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv
    Add::

    # backports
    deb http://backports.debian.org/debian-backports squeeze-backports main

    You must update aptitude's records::

    The options mean:
    sudo apt-get update
    sudo apt-get -t squeeze-backports install nginx # install nginx

    chdir /path/to/your/project
    use your Django project directory as a base
    module project.wsgi
    i.e. the Python wsgi module in your project
    virtualenv /path/to/virtualenv
    the virtualenv
    sudo /etc/init.d/nginx start # start nginx

    There is an alternative to using the `--module` option, by referring instead to the wsgi *file*::
    And now check that the server is serving by visiting it in a web browser. You'll get a "Welcome to nginx!" message.

    wsgi-file /path/to/your/project/project/wsgi.py
    i.e. the system file path to the wsgi.py file
    Add www-data (default nginx user) to the arkestra group, so they can conveniently share folders::

    sudo usermod -a -G arkestra www-data

    Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.
    **************
    Other software
    **************

    Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.
    Version control
    ===============

    ::

    Basic nginx
    ===========
    sudo apt-get install git mercurial # install Git and Mercurial

    Install nginx
    -------------

    The version of Nginx from Debian stable is rather old. We'll install from backports.
    Python
    ======

    ::

    sudo pico /etc/apt/sources.list # edit the sources list
    sudo apt-get install python-virtualenv python-dev # install virtualenv & development version of Python

    sudo apt-get install python-ldap # install ldap support


    Video
    =====

    We install Handbrake and ffmpeg2theora::

    sudo pico /etc/apt/sources.list # edit the sources list (because handbrake isn't otherwise available)

    Add::

    # backports
    deb http://backports.debian.org/debian-backports squeeze-backports main
    #Handbrake
    deb http://www.debian-multimedia.org squeeze main # use this multmedia repository

    Then::

    sudo apt-get update
    sudo apt-get install handbrake-cli ffmpeg2theora


    Miscellaneous libraries
    =======================

    ::

    sudo apt-get install libxml2-dev libxslt-dev libjpeg8-dev # install some libraries


    uwsgi
    =====

    Install uwsgi for the system::

    sudo pip install uwsgi

    Django & Arkestra
    =================

    ::

    cd # just make sure we're in ~/
    virtualenv arkestra # the virtual environment that will contain all our Python software
    cd arkestra
    source bin/activate

    pip install -e git+https://github.com/evildmp/Arkestra.git@pip#egg=Arkestra # install Arkestra & dependencies

    It's necessary to keep an eye on the output of the installation procedure - sometimes a server hosting some dependency won't be available, so make sure there are no errors. If necessary, just run it again.

    Install the right versions of Semantic Editor, Django Widgetry and Django Filer::

    pip install -r src/arkestra/REQUIREMENTS.txt # install the things that pip can't do automatically


    *********************
    Test the example site
    *********************

    You can skip to *Set up a production server* if you don't need to test this.

    ::

    cd src/arkestra/example_14
    python manage.py syncdb # answer "no"

    python manage.py reset contenttypes # answer "yes"

    python manage.py loaddata example_database.json
    python manage.py runserver 0.0.0.0:8000

    The admin username and password are both "arkestra"

    Check that media and static files are being served correctly by the runserver; visit:

    Run::
    * /media/media.png
    * /static/static.png

    sudo apt-get -t squeeze-backports install nginx # install nginx
    sudo /etc/init.d/nginx start # start nginx

    And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"
    **************************
    Set up a production server
    **************************

    Configure nginx for your site
    -----------------------------

    Check that your nginx has installed a file at `/etc/nginx/uwsgi_params`. If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params

    Create a file called nginx.conf, and put this in it::

    # nginx.conf
    upstream django {
    # connect to this socket
    # server unix:///tmp/uwsgi.sock; # for a file socket
    server 127.0.0.1:8001; # for a web port socket
    }
    server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name .example.com; # substitute your machine's IP address or FQDN
    charset utf-8;
    #Max upload size
    client_max_body_size 75M; # adjust to taste

    # Django media
    location /media {
    alias /path/to/your/project/project/media; # your Django project's media files
    }
    location /static {
    alias /path/to/your/project/project/static; # your Django project's static files
    }
    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass django;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }
    =============================

    Turn off the default site installed by nginx::

    sudo rm /etc/nginx/sites-enabled/default # we remove the default symlink

    Check that nginx has installed a file at /etc/nginx/uwsgi_params::

    ls /etc/nginx/uwsgi_params

    If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it:

    ::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params


    nginx for the example site
    ==========================

    You can skip to *Set up the School of Medicine site* if you don't need to test this.

    Create a file called nginx.conf in the project directory::

    pico ~/arkestra/src/arkestra/example_14/example_14_nginx.conf

    and put this in it::

    # example_14_nginx.conf
    upstream django {
    # connect to this socket
    server unix:///tmp/arkestra_example.sock; # a file socket
    }

    server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name .example.com; # substitute the machine's IP address or FQDN
    charset utf-8;

    #Max upload size
    client_max_body_size 75M;

    # Django media
    location /media {
    alias /home/arkestra/arkestra/src/arkestra/example_14/example_14/media;
    }

    location /static {
    alias /home/arkestra/arkestra/src/arkestra/example_14/example_14/static;
    }

    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass django;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }

    Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::

    sudo ln -s ~/path/to/your/project/nginx.conf /etc/nginx/sites-enabled/
    sudo ln -s ~/arkestra/src/arkestra/example_14/example_14_nginx.conf /etc/nginx/sites-enabled/

    Install static files::

    Basic nginx test
    ----------------
    python manage.py collectstatic -l

    Restart nginx::

    sudo /etc/init.d/nginx restart
    sudo /etc/init.d/nginx restart

    Check that media files are being served correctly:

    Add an image called media.png to the /path/to/your/project/project/media directory
    * /media/media.png

    Visit
    If this works, you'll know at least that nginx is serving files correctly.

    http://example.com:8000/media/media.png

    If this works, you'll know at least that nginx is serving files correctly.
    uwsgi for the example site
    ==========================

    nginx and uwsgi and test.py
    ===========================
    Create a file called django.ini in the project directory::

    Let's get nginx to speak to the hello world test.py application.
    pico ~/arkestra/src/arkestra/example_14/example_14.ini

    ::
    and put this in it::

    uwsgi --socket :8001 --wsgi-file test.py
    # django.ini file
    [uwsgi]

    This is nearly the same as before, except now we are not using http between uwsgi and nginx, but the (much more efficient) uwsgi protocol, and we're doing it on port 8001. nginx meanwhile will pass what it finds on that port to port 8000. Visit:
    # master
    master = true

    http://example.com:8000/
    # maximum number of processes
    processes = 10

    to check.
    # the socket (use the full path to be safe)
    socket = /tmp/arkestra_example.sock

    Meanwhile, you can try to have a look at the uswgi output at:
    # 666 is very permissive, but this is just for testing
    chmod-socket = 666

    http://example.com:8001/
    # the base directory
    chdir = /home/arkestra/arkestra/src/arkestra/example_14

    but quite probably, it won't work because your browser speaks http, not uwsgi.
    # Django's wsgi file
    module = example_14.wsgi

    Using sockets instead of ports
    ==============================
    # the virtualenv
    home = /home/arkestra/arkestra

    It's better to use Unix sockets than ports - there's less overhead.
    # clear environment on exit
    vacuum = true

    Edit nginx.conf.
    And run uswgi using the file::

    uncomment
    server unix:///tmp/uwsgi.sock;
    comment out
    server 127.0.0.1:8001;
    uwsgi --ini example_14.ini

    and restart nginx.

    Runs uwsgi again::
    Set up the School of Medicine site
    ==================================

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py
    We will use the existing virtualenv, and put the project in ~/.

    Try http://example.com:8000/ in the browser.
    Install django CMS from git (we need to do this because we'll have to merge a couple of patches specific to the School site)::

    If that doesn't work
    --------------------
    # make sure the virtualenv is activated before you do this!
    pip install -e git+https://github.com/divio/django-cms.git@master#egg=django-cms # django CMS 2.3 from github

    Check your nginx error log(/var/log/nginx/error.log). If you see something like::
    cd ~/arkestra/src/django-cms # this is where pip should have put django-cms
    git remote add evildmp https://github.com/evildmp/django-cms.git # add this as a git remote repository
    git fetch evildmp # fetch information about what's available from the remote
    git merge evildmp/menu-speed-up # merge the branch that speeds up the menus for us
    git merge evildmp/pageflags # merge the pageflags branch we need

    connect() to unix:///path/to/your/project/uwsgi.sock failed (13: Permission denied)
    pip install -e git+https://github.com/evildmp/arkestra-publications.git#egg=arkestra_publications # publications and symplectic applications

    then probably you need to manage the permissions on the socket (especially if you are using a file not in /tmp as suggested).
    pip install django_auth_ldap # LDAP module; we need this for admin users

    Try::
    Ignore this for now - we aren't using celery just yet.

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=644 # 666 permissions (very permissive)
    #pip install -e git+https://github.com/celery/celery.git#egg=celery
    #pip install -e git+https://github.com/celery/django-celery.git#egg=django-celery

    or::
    #sudo apt-get install rabbitmq-server

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=664 # 664 permissions (more sensible)

    You may also have to add your user to nginx's group (probably www-data), or vice-versa, so that nginx can read and write to your socket properly.
    Set up the arkestra_medic database and user
    -------------------------------------------

    Running the Django application with uswgi and nginx
    ===================================================
    You'll need a dump of the live database.

    Let's run our Django application::
    Create the database, and import the dumped database file.

    uwsgi --socket /tmp/uwsgi.sock --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv --chmod-socket=664
    Create a MySQL user "<name of database user>" and give it all permissions for the database.

    Now uwsgi and nginx should be serving up your Django application.
    Set up the arkestra_medic project folder
    ----------------------------------------

    The School of Medicine site (settings, templates, CSS etc) is on GitHub. It's publicly-available, and doesn't include:

    a uwsgi .ini file for our Django application
    ============================================
    * sensitive settings (passwords etc)
    * Filer files (images, videos, etc)

    Deactivate your virtualenv::
    ::

    deactivate
    # we'll install this in ~/
    cd

    and install uwsgi system-wide::
    Clone the arkestra_medic project directory::

    sudo pip install uwsgi
    We can put the same options that we used with uwsgi into a file, and then ask uwsgi to run with that file::
    git clone https://github.com/evildmp/arkestra_medic.git

    # django.ini file
    [uwsgi]
    The media directory needs to be group writable, so nginx can save things in it::

    # master
    master = true
    sudo chgrp -R www-data ~/arkestra_medic/arkestra_medic/media
    sudo chmod -R g+w ~/arkestra_medic/arkestra_medic/media

    # maximum number of processes
    processes = 10
    In ~/arkestra_medic/arkestra_medic/settings.py, some settings need to be set or changed - they are marked with the string *change this!* and include:

    # the socket (use the full path to be safe)
    socket = /tmp/uwsgi.sock
    * DATABASES (you created the database and users earlier, so you know what they should be)
    * SECRET_KEY (some long random string)

    # with appropriate permissions - *may* be needed
    # chmod-socket = 664
    Copy media files (from the current School site)::

    # the base directory
    chdir = /path/to/your/project
    scp -r -P 8888 <user>@<fully-qualified-domain-name-of-site>:</path/to/filer/directory> ~/arkestra_medic/arkestra_medic/media/filer

    # Django's wsgi file
    module = project.wsgi

    # the virtualenv
    home = /path/to/virtualenv
    Test it all runs::

    # clear environment on exit
    vacuum = true
    python manage.py runserver 0.0.0.0:8001 # 8001 because nginx may already be using 8000, if you tested it with the example project

    Create a file called arkestra_medic_nginx.conf in the project directory::

    And run uswgi using the file::
    pico ~/arkestra_medic/arkestra_medic_nginx.conf

    uwsgi --ini django.ini
    and put this in it::

    Note:
    # nginx.conf
    upstream arkestra_medic {
    # connect to this socket
    server unix:///tmp/arkestra_medic.sock; # a file socket
    }

    --ini django.ini
    use the specified .ini file
    server {
    # the port your site will be served on
    listen 80;
    # the domain name it will serve for
    server_name .example.com; # substitute the machine's IP address or FQDN
    charset utf-8;

    Test emperor mode
    =================
    #Max upload size
    client_max_body_size 75M;

    uwsgi can run in 'emperor' mode. In this mode it keeps an eye on a directory of uwsgi config files, and spawns instances ('vassals') for each one it finds.
    # Django media
    location /media {
    alias /home/arkestra/arkestra_medic/arkestra_medic/media;
    }

    Whenever a config file is amended, the emperor will automatically restart the vassal.
    location /static {
    alias /home/arkestra/arkestra_medic/arkestra_medic/static;
    }

    ::
    # Finally, send all non-media requests to the Django server.
    location / {
    uwsgi_pass arkestra_medic;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }

    Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::

    sudo ln -s ~/arkestra_medic/arkestra_medic_nginx.conf /etc/nginx/sites-enabled/

    Install static files::

    cd ~/arkestra_medic
    python manage.py collectstatic -l

    Restart nginx::

    sudo /etc/init.d/nginx restart

    Check that media files are being served correctly:

    * /media/media.png

    If this works, you'll know at least that nginx is serving files correctly.

    Create a file called arkestra_medic_uwsgi.ini in the project directory::

    pico ~/arkestra_medic/arkestra_medic_uwsgi.ini

    and put this in it::

    # symlink from the default config directory to your config file
    sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/
    # django.ini file
    [uwsgi]

    # run the emperor as root
    sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
    # master
    master = true

    The options mean:
    # maximum number of processes
    processes = 10

    emperor /etc/uwsgi/vassals
    look there for vassals (config files)
    uid www-data
    run as www-data once we've started
    gid www-data
    run as www-data once we've started
    # the socket (use the full path to be safe)
    socket = /tmp/arkestra_medic.sock

    Check the site; it should be running.
    # 666 is very permissive, but you can use this if necessary just for testing
    # chmod-socket = 666

    Make uwsgi startup when the system boots
    ========================================
    # the base directory
    chdir = /home/arkestra/arkestra_medic

    # Django's wsgi file
    module = arkestra_medic.wsgi

    # the virtualenv
    home = /home/arkestra/arkestra

    # clear environment on exit
    vacuum = true

    Deactivate your virtualenv::

    deactivate

    And run uswgi using the file::

    sudo uwsgi --ini arkestra_medic_uwsgi.ini --uid www-data --gid www-data


    Edit /etc/rc.local::

    sudo pico /etc/rc.local

    and add::

    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master

    before the line `"exit 0"`.

    ::

    The last step is to make it all happen automatically at system startup time.
    # symlink from the default config directory to your config file
    sudo mkdir -p /etc/uwsgi/vassals/
    sudo ln -s ~/arkestra_medic/arkestra_medic_uwsgi.ini /etc/uwsgi/vassals/

    Edit /etc/rc.local and add::
    Just for now, because the current live server is old, we need to change a couple of items in the database - adjust contenttypes table and filer_video table

    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
    * django_content_type id 212, change app_name from filer to video
    * rename filer_video table to video_video

    before the line "exit 0".
    Reboot the server and it should work at startup::

    And that should be it!
    sudo reboot
  4. @superdmp superdmp revised this gist Jul 12, 2012. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -338,7 +338,7 @@ Whenever a config file is amended, the emperor will automatically restart the va
    sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/

    # run the emperor as root
    sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data
    sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master

    The options mean:

    @@ -358,7 +358,7 @@ The last step is to make it all happen automatically at system startup time.

    Edit /etc/rc.local and add::

    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data
    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master

    before the line "exit 0".

  5. @superdmp superdmp revised this gist Jul 12, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -298,7 +298,7 @@ We can put the same options that we used with uwsgi into a file, and then ask uw
    processes = 10

    # the socket (use the full path to be safe)
    socket = /path/to/your/project/uwsgi.sock
    socket = /tmp/uwsgi.sock

    # with appropriate permissions - *may* be needed
    # chmod-socket = 664
  6. @superdmp superdmp revised this gist Jul 12, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -179,7 +179,7 @@ Create a file called nginx.conf, and put this in it::
    # Finally, send all non-media requests to the Django server.
    location / {
    proxy_pass django;
    uwsgi_pass django;
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }
  7. @superdmp superdmp revised this gist Jul 12, 2012. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -145,7 +145,7 @@ And now check that the server is serving by visiting it in a web browser on port
    Configure nginx for your site
    -----------------------------

    Copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory. nginx will need this. Easiest way::
    Check that your nginx has installed a file at `/etc/nginx/uwsgi_params`. If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params

    @@ -180,7 +180,7 @@ Create a file called nginx.conf, and put this in it::
    # Finally, send all non-media requests to the Django server.
    location / {
    proxy_pass django;
    include /path/to/your/project/uwsgi_params; # the uwsgi_params file you installed
    include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
    }
    }

  8. @superdmp superdmp revised this gist Jul 12, 2012. 1 changed file with 7 additions and 1 deletion.
    8 changes: 7 additions & 1 deletion gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -104,10 +104,16 @@ The options mean:
    chdir /path/to/your/project
    use your Django project directory as a base
    module project.wsgi
    i.e. /path/to/your/project/project/wsgi.py
    i.e. the Python wsgi module in your project
    virtualenv /path/to/virtualenv
    the virtualenv

    There is an alternative to using the `--module` option, by referring instead to the wsgi *file*::

    wsgi-file /path/to/your/project/project/wsgi.py
    i.e. the system file path to the wsgi.py file


    Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.

    Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.
  9. @superdmp superdmp revised this gist Jul 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -35,7 +35,7 @@ Note that I'm also assuming a Django 1.4 project structure, in which you see pat

    /path/to/your/project/project/

    (i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier example.
    (i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.

    It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.

  10. @superdmp superdmp revised this gist Jul 11, 2012. 1 changed file with 8 additions and 10 deletions.
    18 changes: 8 additions & 10 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -72,10 +72,10 @@ Run::

    The options mean:

    --http :8000
    http :8000
    use protocol http, port 8000

    --wsgi-file test.py
    wsgi-file test.py
    load the specified file

    This should serve a hello world message directly to the browser on port 8000. Visit::
    @@ -101,11 +101,11 @@ Now run it using uwsgi::

    The options mean:

    --chdir /path/to/your/project
    chdir /path/to/your/project
    use your Django project directory as a base
    --module project.wsgi
    module project.wsgi
    i.e. /path/to/your/project/project/wsgi.py
    --virtualenv /path/to/virtualenv
    virtualenv /path/to/virtualenv
    the virtualenv

    Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.
    @@ -336,13 +336,11 @@ Whenever a config file is amended, the emperor will automatically restart the va

    The options mean:

    --emperor /etc/uwsgi/vassals
    emperor /etc/uwsgi/vassals
    look there for vassals (config files)

    --uid www-data
    uid www-data
    run as www-data once we've started

    --gid www-data
    gid www-data
    run as www-data once we've started

    Check the site; it should be running.
  11. @superdmp superdmp created this gist Jul 11, 2012.
    361 changes: 361 additions & 0 deletions gistfile1.rst
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,361 @@
    ******************************
    Set up Django, nginx and uwsgi
    ******************************

    Steps with explanations to set up a server using:

    * virtualenv
    * Django
    * nginx
    * uwsgi

    Concept
    =======

    nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.

    That's uwsgi's job. uwsgi will create a Unix socket, and serve responses to nginx via the uwsgi protocol - the socket passes data in both directions::

    the outside world <-> nginx <-> the socket <-> uwsgi

    Before you start
    ================

    virtualenv
    ----------

    Make sure you are in a virtualenv - you will install a system-wide uwsgi later.

    Django
    ------

    I am assuming Django 1.4. It automatically creates a wsgi module when you create a project. If you're using an earlier version, you will have to find a Django wsgi module for your project.

    Note that I'm also assuming a Django 1.4 project structure, in which you see paths like::

    /path/to/your/project/project/

    (i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier example.

    It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.

    About the domain and port
    -------------------------

    I'll call your domain example.com. Substitute your own FQDN or IP address.

    Throughout, I'm using port 8000. You can use whatever port you want of course, but I have chosen this one so it doesn't conflict with anything a web server might be doing already.

    Basic uwsgi intallation and configuration
    =========================================

    Install uwsgi
    -------------

    ::

    pip install uwsgi

    Basic test
    ----------

    Create a file called test.py::

    # test.py
    def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return "Hello World"

    Run::

    uwsgi --http :8000 --wsgi-file test.py

    The options mean:

    --http :8000
    use protocol http, port 8000

    --wsgi-file test.py
    load the specified file

    This should serve a hello world message directly to the browser on port 8000. Visit::

    http://example.com:8000

    to check.

    Test your Django project
    ------------------------

    Now we want uwsgi to do the same thing, but to run a Django site instead of the test.py module.

    But first, make sure that your project actually works! Now you need to be in your Django project directory.

    ::

    python manage.py runserver 0.0.0.0:8000

    Now run it using uwsgi::

    uwsgi --http :8000 --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv

    The options mean:

    --chdir /path/to/your/project
    use your Django project directory as a base
    --module project.wsgi
    i.e. /path/to/your/project/project/wsgi.py
    --virtualenv /path/to/virtualenv
    the virtualenv

    Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.

    Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.

    Basic nginx
    ===========

    Install nginx
    -------------

    The version of Nginx from Debian stable is rather old. We'll install from backports.

    ::

    sudo pico /etc/apt/sources.list # edit the sources list

    Add::

    # backports
    deb http://backports.debian.org/debian-backports squeeze-backports main

    Run::

    sudo apt-get -t squeeze-backports install nginx # install nginx
    sudo /etc/init.d/nginx start # start nginx

    And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"

    Configure nginx for your site
    -----------------------------

    Copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory. nginx will need this. Easiest way::

    wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params

    Create a file called nginx.conf, and put this in it::

    # nginx.conf
    upstream django {
    # connect to this socket
    # server unix:///tmp/uwsgi.sock; # for a file socket
    server 127.0.0.1:8001; # for a web port socket
    }
    server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name .example.com; # substitute your machine's IP address or FQDN
    charset utf-8;
    #Max upload size
    client_max_body_size 75M; # adjust to taste

    # Django media
    location /media {
    alias /path/to/your/project/project/media; # your Django project's media files
    }
    location /static {
    alias /path/to/your/project/project/static; # your Django project's static files
    }
    # Finally, send all non-media requests to the Django server.
    location / {
    proxy_pass django;
    include /path/to/your/project/uwsgi_params; # the uwsgi_params file you installed
    }
    }

    Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::

    sudo ln -s ~/path/to/your/project/nginx.conf /etc/nginx/sites-enabled/

    Basic nginx test
    ----------------

    Restart nginx::

    sudo /etc/init.d/nginx restart

    Check that media files are being served correctly:

    Add an image called media.png to the /path/to/your/project/project/media directory

    Visit

    http://example.com:8000/media/media.png

    If this works, you'll know at least that nginx is serving files correctly.

    nginx and uwsgi and test.py
    ===========================

    Let's get nginx to speak to the hello world test.py application.

    ::

    uwsgi --socket :8001 --wsgi-file test.py

    This is nearly the same as before, except now we are not using http between uwsgi and nginx, but the (much more efficient) uwsgi protocol, and we're doing it on port 8001. nginx meanwhile will pass what it finds on that port to port 8000. Visit:

    http://example.com:8000/

    to check.

    Meanwhile, you can try to have a look at the uswgi output at:

    http://example.com:8001/

    but quite probably, it won't work because your browser speaks http, not uwsgi.

    Using sockets instead of ports
    ==============================

    It's better to use Unix sockets than ports - there's less overhead.

    Edit nginx.conf.

    uncomment
    server unix:///tmp/uwsgi.sock;
    comment out
    server 127.0.0.1:8001;

    and restart nginx.

    Runs uwsgi again::

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py

    Try http://example.com:8000/ in the browser.

    If that doesn't work
    --------------------

    Check your nginx error log(/var/log/nginx/error.log). If you see something like::

    connect() to unix:///path/to/your/project/uwsgi.sock failed (13: Permission denied)

    then probably you need to manage the permissions on the socket (especially if you are using a file not in /tmp as suggested).

    Try::

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=644 # 666 permissions (very permissive)

    or::

    uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=664 # 664 permissions (more sensible)

    You may also have to add your user to nginx's group (probably www-data), or vice-versa, so that nginx can read and write to your socket properly.

    Running the Django application with uswgi and nginx
    ===================================================

    Let's run our Django application::

    uwsgi --socket /tmp/uwsgi.sock --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv --chmod-socket=664

    Now uwsgi and nginx should be serving up your Django application.


    a uwsgi .ini file for our Django application
    ============================================

    Deactivate your virtualenv::

    deactivate

    and install uwsgi system-wide::

    sudo pip install uwsgi
    We can put the same options that we used with uwsgi into a file, and then ask uwsgi to run with that file::

    # django.ini file
    [uwsgi]

    # master
    master = true

    # maximum number of processes
    processes = 10

    # the socket (use the full path to be safe)
    socket = /path/to/your/project/uwsgi.sock

    # with appropriate permissions - *may* be needed
    # chmod-socket = 664

    # the base directory
    chdir = /path/to/your/project

    # Django's wsgi file
    module = project.wsgi

    # the virtualenv
    home = /path/to/virtualenv

    # clear environment on exit
    vacuum = true


    And run uswgi using the file::

    uwsgi --ini django.ini

    Note:

    --ini django.ini
    use the specified .ini file

    Test emperor mode
    =================

    uwsgi can run in 'emperor' mode. In this mode it keeps an eye on a directory of uwsgi config files, and spawns instances ('vassals') for each one it finds.

    Whenever a config file is amended, the emperor will automatically restart the vassal.

    ::

    # symlink from the default config directory to your config file
    sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/

    # run the emperor as root
    sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data

    The options mean:

    --emperor /etc/uwsgi/vassals
    look there for vassals (config files)

    --uid www-data
    run as www-data once we've started

    --gid www-data
    run as www-data once we've started

    Check the site; it should be running.

    Make uwsgi startup when the system boots
    ========================================

    The last step is to make it all happen automatically at system startup time.

    Edit /etc/rc.local and add::

    /usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data

    before the line "exit 0".

    And that should be it!