Skip to content

Instantly share code, notes, and snippets.

@Atem18
Forked from evildmp/gist:3094281
Last active April 19, 2024 11:18
Show Gist options
  • Save Atem18/4696071 to your computer and use it in GitHub Desktop.
Save Atem18/4696071 to your computer and use it in GitHub Desktop.
Tutorial to seting up a django website in production.

Set up Django, Nginx and Gunicorn in a Virtualenv controled by Supervisor

Steps with explanations to set up a server using:

  • Virtualenv
  • Virtualenvwrapper
  • Django
  • Gunicorn
  • Nginx
  • Supervisor

Concept

Nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.

That's Gunicorn's job. Gunicorn will create a Unix socket, and serve responses to nginx via the wsgi protocol - the socket passes data in both directions:

The outside world <-> Nginx <-> The socket <-> Gunicorn

All this family will live into a Virtualenv. Already wondered why Virtualenv is so useful when you develop Pythons's applications? Continue to read and you will understand.

Before you start

1. Virtualenv

If you don't already use a Virtualenv, you really should consider using it. In a nutshell, the fact is that your operating system have a lot of core modules which depends of Python. That rule is especially right if you use Ubuntu. So, do you really need to broke up something existant to create somethin new? Well, some guys could answer you "Yes". I'll teach you how to say "No" to them ! Virtualenv will create a dedicated virtual environment with his python binary as well as his own modules. With that method, your Virtualenv will be isolated from the rest of the system.

1.5 Pip (optional, but higly recommanded)

If you use Pip, and I clearly recommand to use it, Virtualenv will simplify your life, because if you play well, you only have to do

pip freeze > requirements.txt

to print a list of all the modules needed by your project into a file. When you will deploy it, you will be able to install all those module with a

pip install -r requirements.txt

2. Virtualenvwrapper

What ? I didn't explained how to use Virtualenv? Well, indeed I lied. We won't be using Virtualenv directly. We will use a wrapper. Virtualenvwrapper to be right. Original, isn't it? Virtualenvwrapper will allow you to easily create virtualenv and switching between them.

Let's get started ! To install virtualenvwrapper, install it with Pip, apt-get, whatever:

pip install virtualenvwrapper

Virtualenvwrapper is now installed? Let's play with him. First, you will need to modify your .bashrc with your favourite text editor. Add this lines at the and of your .bashrc

export WORKON_HOME=~/.virtualenvs
export PROJECT_HOME=/path/to/your/project/home
source /usr/local/bin/virtualenvwrapper.sh

WORKON_HOME is required but PROJECT_HOME is needless. Don't forget to always put ALL the export's directives before sourcing your file. Seems legit for you? Here, have my like !

Now, let's activate our .bashrc. If you are a UNIX guru, you have problably already did it before I explained it. If not, here is the command

source .bashrc

If all is good, your stdout will be sourced byt the output of the Virtualenvwrapper creating script. You can verify it by searching a .virtualenvs folder is your home directory.

Now, let's ceate our virtualenv for our project. You can name it whatever your want and even if you don't remember, just go into your .virtualenvs folder and the name of your virtualenv will be one of the folder. The command for creating one is

mkvirtualenv nameofyourproject

The command for working on a specified virtualenv is

workon nameofyourproject

The command for deactivate a specified virtualenv is

deactivate nameofyourproject

And finally the command to remove a specified virtualenv is

rmvirtualenv nameofyourproject

That's all the command you need to remember. If the workon command has successfully created a virtualenv, your should see the name of your project between parentheses in the left of your username@nameofyourcomputer. Something like that

(nameofyourproject)username@nameofyourcomputer

All is good, Okey, let's move to the next of the story.

! REMEMBER TO DO ALL THE NEXT PART IN YOUR VIRTUALENV. ONLY SUPERVISOR WILL BE INSTALLED OUTSIDE YOUR VIRTUALENV !

1. Django

I'm assuming you are using Django 1.4.x. Everything should be fine from our right for Django 1.5. Anayway, if you created your project in 1.4, it should have automatically created a wsgi module. If you're using an earlier version, you will have to find a Django wsgi module for your project. Here is mine modified for exemple

 import os
 import sys


 root = os.path.join(os.path.dirname(__file__), '..')
 sys.path.insert(0, root)


 os.environ['DJANGO_SETTINGS_MODULE'] = 'nameofyourproject.settings'


 import django.core.handlers.wsgi
 application = django.core.handlers.wsgi.WSGIHandler()

 # This application object is used by any WSGI server configured to use this
 # file. This includes Django's development server, if the WSGI_APPLICATION
 # setting points here.
 from django.core.wsgi import get_wsgi_application
 application = get_wsgi_application()

# Apply WSGI middleware here.
# from helloworld.wsgi import HelloWorldApplication
# application = HelloWorldApplication(application)

Note that I'm also assuming a Django 1.4 project structure, in which you see paths like:

/path/to/your/project/project/

(i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.

It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.

About the domain and port

I'll call your domain domain.tld. Substitute your own FQDN or IP address.

Throughout, I'm using the port (:8000) for tests because it is the same port as Django's dev web server. The default port for http (:80) will be used for real deployment. You can use whatever port you want of course, but if you don't know what NAT is, don't do it.

Basic Gunicorn installation and configuration

Install Gunicorn

As we said earlier, Gunicorn will serve the core of our application, just like the Django's development web server would do it. Let's install it

pip install gunicorn

Basic test

Create a file called myapp.py:

def app(environ, start_response):
    data = "Hello, World!\n"
    start_response("200 OK", [
        ("Content-Type", "text/plain"),
        ("Content-Length", str(len(data)))
    ])
return iter([data])

Run:

gunicorn -w 4 myapp:app

This should serve a hello world message directly to the browser on port 8000. Visit:

http://127.0.0.1:8000

to check.

Test your Django project

Now we want gunicorn to do the same thing, but to run a Django site instead of the test.py module.

But first, make sure that your project actually works! Now you need to be in your Django project directory.

python manage.py runserver 0.0.0.0:8000

Now run it using gunicorn:

gunicorn nameofyourapp.wsgi:app

Point your browser at the server; if the site appears, it means gunicorn can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.

Now normally we won't have the browser speaking directly to gunicorn: nginx will be the go-between.

Basic nginx

Install nginx

The version of Nginx from Debian stable and Ubuntu is rather old. We'll install from backports for Debian and PPA for Ubuntu.

Debian:

sudo nano /etc/apt/sources.list     # edit the sources list

Add:

# backports
deb http://backports.debian.org/debian-backports squeeze-backports main

Run:

sudo apt-get -t squeeze-backports install nginx # install nginx

For Ubuntu:

sudo -s
nginx=stable # use nginx=development for latest development version
add-apt-repository ppa:nginx/$nginx
apt-get update
apt-get install nginx

For both:

sudo /etc/init.d/nginx start    # start nginx

And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"

Configure nginx for your site

Create a file called nginx.conf, and put this in it:

server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name .domain.tld ip.adress;   # substitute by your FQDN and machine's IP address
    charset     utf-8;

    #Max upload size
    client_max_body_size 75M;   # adjust to taste

    # Django media
    location /media  {
        alias /var/www/path/to/your/project/media;      # your Django project's media files
    }

    location /assets {
        alias /var/www/path/to/your/project/static;     # your Django project's static files
    }

    # Finally, send all non-media requests to the Django server.
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Move your file to /etc/nginx/sites-enabled so nginx can see it:

sudo mv /path/to/your/project/nginx.conf /etc/nginx/sites-enabled/

Basic nginx test

Restart nginx:

sudo /etc/init.d/nginx restart

Check that media files are being served correctly:

Add an image called media.png to the /path/to/your/project/project/media directory

Visit

http://domain.tld:80/media/media.png

If this works, you'll know at least that nginx is serving files correctly.

Running the Django application with gunicorn and nginx

Let's run our Django application:

gunicorn nameofyourapp.wsgi:app

Now gunicorn and nginx should be serving up your Django application.

Make gunicorn startup when the system boots

The last step is to make it all happen automatically at system startup time.

To do this, we will install a programm called supervisor:

sudo pip install supervisor

Sudo in front of your command is important, because it means that our supervisor programm will be installed in the system and not in a virtualenv.

Now, we will create a bash file to execute some commands such as activate our virtualenv and browsing to our directory. So create a nameoryourproject.sh file:

#!/bin/bash
WORKING_DIR=/path/to/your/project
ACTIVATE_PATH=/path/to/your/virtualenv/bin/activate
cd ${WORKING_DIR}
source ${ACTIVATE_PATH}
exec $@

Next, we create a file named supervisord.conf and we put it in /etc/

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Note: shell expansion ("~" or "$HOME") is not supported.  Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
chmod=0700                 ; socket file mode (default 0700)
chown=root:root       ; socket file uid:gid owner
;username=noname              ; (default is no username (open server))
;password=noname               ; (default is no password (open server))

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=127.0.0.1:9001        ; (ip_address:port specifier, *:port for all iface)
;username=noname              ; (default is no username (open server))
;password=noname               ; (default is no password (open server))

[supervisord]
logfile=/var/log/supervisord/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=true               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
;umask=022                   ; (process file creation umask;default 022)
user=root                 ; (default is current user, required if root)
;identifier=supervisor       ; (supervisord identifier, default is 'supervisor')
;directory=/tmp              ; (default is not to cd during start)
;nocleanup=true              ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp            ; ('AUTO' child log dir, default $TEMP)
;environment=KEY=value       ; (key value pairs to add to environment)
;strip_ansi=false            ; (strip ansi escape codes in logs; def. false)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=www-data              ; should be same as http_username if set
;password=www-data                ; should be same as http_password if set
;prompt=mysupervisor         ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history  ; use readline history if available

; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.

;[program:theprogramname]
;command=/bin/cat              ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
;directory=/tmp                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=999                  ; the relative start priority (default 999)
;autostart=true                ; start at supervisord start (default: true)
;autorestart=unexpected        ; whether/when to restart (default: unexpected)
;startsecs=1                   ; number of secs prog must stay running (def. 1)
;startretries=3                ; max # of serial start failures (default 3)
;exitcodes=0,2                 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
;killasgroup=false             ; SIGKILL the UNIX process group (def false)
;user=chrism                   ; setuid to this UNIX account to run the program
;redirect_stderr=true          ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10     ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;environment=A=1,B=2           ; process environment additions (def no adds)
;serverurl=AUTO                ; override serverurl computation (childutils)

; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.

;[eventlistener:theeventlistenername]
;command=/bin/eventlistener    ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
;events=EVENT                  ; event notif. types to subscribe to (req'd)
;buffer_size=10                ; event buffer queue size (default 10)
;directory=/tmp                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=-1                   ; the relative start priority (default -1)
;autostart=true                ; start at supervisord start (default: true)
;autorestart=unexpected        ; whether/when to restart (default: unexpected)
;startsecs=1                   ; number of secs prog must stay running (def. 1)
;startretries=3                ; max # of serial start failures (default 3)
;exitcodes=0,2                 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
;killasgroup=false             ; SIGKILL the UNIX process group (def false)
;user=chrism                   ; setuid to this UNIX account to run the program
;redirect_stderr=true          ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (default 10)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups        ; # of stderr logfile backups (default 10)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;environment=A=1,B=2           ; process environment additions
;serverurl=AUTO                ; override serverurl computation (childutils)

; The below sample group section shows all possible group values,
; create one or more 'real' group: sections to create "heterogeneous"
; process groups.

;[group:thegroupname]
;programs=progname1,progname2  ; each refers to 'x' in [program:x] definitions
;priority=999                  ; the relative start priority (default 999)

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

;[include]
;files = relative/directory/*.ini
[program:nameofyourprogram]
directory = /path/of/your/project/
user = www-data
command = /path/to/your/nameofyourproject.sh gunicorn nameofyourproject.wsgi:application
stdout_logfile = /var/log/supervisord/access.log
stderr_logfile = /var/log/supervisord/error.log

After, we will create a file which will ask supervisord to boot at the startup.

Create a file named supervisord:

# Supervisord auto-start
#
# description: Auto-starts supervisord
# processname: supervisord
# pidfile: /var/run/supervisord.pid

SUPERVISORD=/usr/local/bin/supervisord
SUPERVISORD_ARGS='-c /etc/supervisord.conf'
SUPERVISORCTL=/usr/local/bin/supervisorctl

case $1 in
start)
    echo -n "Starting supervisord: "
    $SUPERVISORD $SUPERVISORD_ARGS
    echo
    ;;
stop)
    echo -n "Stopping supervisord: "
    $SUPERVISORCTL shutdown
    echo
    ;;
restart)
    echo -n "Stopping supervisord: "
    $SUPERVISORCTL shutdown
    echo
    echo -n "Starting supervisord: "
    $SUPERVISORD $SUPERVISORD_ARGS
    echo
    ;;
esac

Make it executable

chmod +x supervisord

Move it to /etc/init.d

mv supervisord /etc/init.d/

Make it boot at the start

sudo update-rc.d supervisord defaults

Reboot your system and your website should live through the ages.

@aishwarydhare
Copy link

can i use it for running django web-app on RaspberryPi

@gamesbook
Copy link

Any updates planned for the newer versions of Django; 1.8 upwards?

@adykumar
Copy link

This is beautiful! Helped me understand with great clarity!
On a related note, if you need help improving the explanation (spelling errors etc) I can help. Not that its unclear in any way.
Kudos!

@mishrade
Copy link

mishrade commented Aug 27, 2017

Hi, This was amazing. I was able to follow the steps exactly and deploy my Django project. I am using gunicorn with nginx.
I wanted to run my application on Https. how can I achieve that?

@dornlabs
Copy link

That you great work here

@EvanZ
Copy link

EvanZ commented Jan 9, 2018

Good writeup. I have recently set up the same stack, but with the addition of Postgres. I would suggest adding that here, if possible.

@mhsiddiqui
Copy link

How can I run it on https?

@Atem18
Copy link
Author

Atem18 commented Dec 5, 2018

Hi,

I no longer use Django and exclusively use docker.

For Postgres, it's not in the original scope but if you want to add it, please do.

For HTTPS, my advice would be to use Treafik : https://docs.traefik.io/

If you use Docker, then you know how to do it

If you want to launch it with supervisor :

[program:traefik]
process_name=%(program_name)s
command=/usr/local/bin/%(program_name)s -c /etc/%(program_name)s/%(program_name)s.toml
directory=/etc/%(program_name)s
user=root
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/%(program_name)s/%(program_name)s-stdout.log
stdout_logfile_maxbytes=100MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB

@jxub
Copy link

jxub commented Mar 6, 2019

Why specifically use supervisor and not systemd/init... ?

@fromthestone
Copy link

I am learning Python for a few weeks, this is very helpful.
Thanks

@Atem18
Copy link
Author

Atem18 commented Jun 6, 2019

@jxub Because systemd was not available at the time I wrote this guide and init is old and obscure. But yeah now if you want, you can and probably should use systemd to manage your services(Django, Traefik, Nginx, etc.).

Do note that some people are using supervisor to launch multiples services in a Docker container. I don't agree with it because it breaks the original docker bahaviour of one container equal one app, but in my previous job we had no choice but to run our apps besides rsyslogd to forward the logs and the two apps were managed by supervisor.

@Atem18
Copy link
Author

Atem18 commented Jun 6, 2019

Also, I am writing a crypto currency trading bot currently.

I am using Django 2.2 with Django rest framework as backend to have a REST API.

VueJS 2 is the frontend that will query the API.

Database is a PostgreSQL with TimescaleDB enabled for some tables.

They will run in Docker containers and traefik will redirect you to either the API or the frontend based on the domain.

I plan to make a tutorial (either on Gist or on my website) on how to run all of that in production when i have something to show.

So stay tuned.

@ce0la
Copy link

ce0la commented Jan 16, 2020

Also, I am writing a crypto currency trading bot currently.

I am using Django 2.2 with Django rest framework as backend to have a REST API.

VueJS 2 is the frontend that will query the API.

Database is a PostgreSQL with TimescaleDB enabled for some tables.

They will run in Docker containers and traefik will redirect you to either the API or the frontend based on the domain.

I plan to make a tutorial (either on Gist or on my website) on how to run all of that in production when i have something to show.

So stay tuned.

Please, I have been having trouble deploying my DRF APIs for some days and though your article is great, I would like to see the new one you planned to write. Please, also leave your social media handles so I can connect when I need help related to APIs; it is my first time writing them.

@Atem18
Copy link
Author

Atem18 commented Jan 16, 2020

@ce0la I have been busy with others projects but I will soon continue that one.
But instead of Django, I am now using https://github.com/tiangolo/fastapi because I want to use websockets and the support of websockets in django is lacking.
Also, I will be using Kubernetes to deploy the applications.

If you want to contact me, all the infos are here : https://www.atemlire.io/wiki/

@Atem18
Copy link
Author

Atem18 commented Nov 23, 2020

Hi guys,

Here is a small article I wrote to help you create a small and secure Linux container for your applications.

Even if you don't use or clients don't use containers in production, it can be useful to generate a binary of your application using Pyinstaller that you can then distribute to your clients with other ways.

Here is the link for those interested: https://www.atemlire.io/how-to-create-a-small-and-secure-container-for-your-python-applications/

If you have any feedback, please let me know !

@Atem18
Copy link
Author

Atem18 commented Dec 3, 2020

Also for the people using Kubernetes, I maintain some charts.
You may be interested by this one : https://github.com/Atem18/helm-charts/tree/main/charts/nautilus-api

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment