The Gunicorn "Green Unicorn" is a Python Web Server Gateway Interface (WSGI) HTTP server. Recommended to use with nginx.
gunicorn -v
http://docs.gunicorn.org/en/stable/install.html
An example configuration file: gunicorn_config.py
import multiprocessing
chdir = '/home/miranda/workspace/CodeBench/'
bind = "0.0.0.0:8000"
# bind = 'unix:/run/gunicorn/socket'
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'eventlet'
accesslog = 'gunicorn.access'
errorlog = 'gunicorn.error'
reload = True # Restart workers when code changes.Use cpu_count() * 2 + 1 gunicorn workers.
You can check number of cores with the following linux command:
nproc
This can be changed up and down at runtime using signals. See gunicorn docs for details.
gunicorn -c gunicorn_config.py myproject.wsgi:application
e.g. :
gunicorn -c gunicorn_config.py codebench.wsgi:application
We passed Gunicorn a module by specifying the relative directory path to Django's wsgi.py file, which is the entry point to our application, using Python's module syntax. Inside of this file, a function called application is defined, which is used to communicate with the application. To learn more about the WSGI specification, click
here.
TODO: reload, Security(limit_request_field_size) etc.
See documentation for more details.
This setting is intended for development.
A tool that is starting to be common on linux systems is Systemd. Below are configurations files and instructions for using systemd to create a unix socket for incoming Gunicorn requests. Systemd will listen on this socket and start gunicorn automatically in response to traffic. Later in this section are instructions for configuring Nginx to forward web traffic to the newly created unix socket.
/etc/systemd/system/gunicorn.service
[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target
[Service]
PIDFile=/home/miranda/run/gunicorn/pid
User=miranda
Group=www-data
RuntimeDirectory=gunicorn
WorkingDirectory=/home/miranda/workspace/CodeBench/deploy
ExecStart=/root/.virtualenvs/codebench/bin/gunicorn --pid /home/miranda/run/gunicorn/pid \
-c gunicorn_config.py \
codebench.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=always
RestartSec=3
PrivateTmp=true
[Install]
WantedBy=multi-user.target
We'll let the www-data group which Nginx belongs to be the group owners.
RuntimeDirectory is the a subdirectory relative to WorkingDirectory.
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
We need to give the path to the Gunicorn executable, which is stored within our virtual environment. It can be find out by:
type gunicorn
We will tell it to use a Unix socket instead of a network port to communicate with Nginx, since both services will be running on this server. This is more secure and faster.
You can add any other configuration for Gunicorn here as well. For instance, we can specify a config file via -c. WorkingDirectory can specify a directory for your script to execute, we have put the gunicorn_config.py in this directory.
Read more here
/etc/systemd/system/gunicorn.socket
[Unit]
Description=gunicorn socket
[Socket]
ListenStream=/home/miranda/run/gunicorn/socket
[Install]
WantedBy=sockets.target
todo: not working at the moment, using /home/miranda/run/gunicorn instead.
/etc/tmpfiles.d/gunicorn.conf
d /run/gunicorn 0755 miranda www-data -
systemd-tmpfiles uses the configuration files to describe the creation, cleaning and removal of volatile and temporary files and directories which usually reside in directories such as /run or /tmp.
Type Path Mode UID GID Age Argument
d /run/gunicorn 0755 miranda www-data -
See documentation here.
Create a directory. The mode and ownership will be adjusted if specified and the directory already exists. Contents of this directory are subject to time based cleanup if the time argument is specified.
Next enable the socket so it autostarts at boot:
systemctl enable gunicorn.socket
Either reboot, or start the services manually:
systemctl start gunicorn.socket
After running curl --unix-socket /run/gunicorn/socket http, Gunicorn should start and you should see some HTML from your server in the terminal.
- If not, try check:
systemctl status gunicorn
-
Check error log
-
To be sure that gunicorn is refreshed, run these commands:
pkill gunicorn
systemctl daemon-reload
systemctl start gunicorn
- Version related problem:
pip install "eventlet==0.20.0" --force-reinstall
- Ensure systemd services restart on failure
/etc/systemd/system/gunicorn.service
Restart=always
RestartSec=3
In one terminal
watch "ps -ef|grep gunicorn"
Keep the above open, open another terminal, try to kill it:
pkill gunicorn
You should see the process restart.
You must now configure your web proxy to send traffic to the new Gunicorn socket. Edit your nginx.conf to include the following:
proxy_pass http://unix:/run/gunicorn/socket;
Read more here or read the section below about Nginx.
Now make sure you enable the nginx service so it automatically starts at boot:
systemctl enable nginx.service
Either reboot, or start Nginx with the following command:
systemctl start nginx
It does a small subset tasks which can be done with systemd (or upstart in earlier version of system).
One advantage of supervisor is that it is largely platform agnostic,so you don't have to worry about whether your distro has upstart/systemd/whatever. Plus, supervisor is a bit easier to configure and get running correctly when all you want to do is keep an app running forever.
Before installation make sure port 80 is not being used.
netstat -tulpn | grep :80
apt install nginx-core
An example configuration file with Nginx:
http://docs.gunicorn.org/en/stable/deploy.html
All nginx configuration files are located in the /etc/nginx/ directory. The primary configuration file is /etc/nginx/nginx.conf. Note, This is where the files will be located if you install nginx from the package manager.
python manage.py collectstatic
Try to avoid too many chained inclusions (i.e., including a file that itself includes a file, etc.) Keep it to one or two levels of inclusion if possible, for readability purposes. You can include all files in a certain directory. Or to be more specific, you can include all .conf files in a directory:
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
This allows for server block configurations to be loaded in from separate files found in the sites-enabled sub-directory. Usually these are symlinks to files stored in /etc/nginx/sites-available/. By using symlinks you can quickly enable or disable a virtual server while preserving its configuration file.
Linux command, create a link to TARGET with the name LINK_NAME:
ln -sf TARGET LINK_NAME
-s, --symbolic: make symbolic links instead of hard links
-f, --force: remove existing destination files
-
Nginx gets rid of any empty headers. There is no point of passing along empty values to another server; it would only serve to bloat the request.
-
Nginx, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request. If you wish to have Nginx interpret these as valid, you can set the
underscores_in_headersdirective to "on", otherwise your headers will never make it to the backend server. -
The "Host" header is re-written to the value defined by the
$proxy_hostvariable. This will be the IP address or name and port number of the upstream, directly as defined by theproxy_passdirective. -
The "Connection" header is changed to "close". This header is used to signal information about the particular connection established between two parties. In this instance, Nginx sets this to "close" to indicate to the upstream server that this connection will be closed once the original request is responded to. The upstream should not expect this connection to be persistent.
-
$host: This variable is set, in order of preference to: the host name from the request line itself, the "Host" header from the client request, or the server name matching the request. There are other common values for the "Host" header: like
$proxy_hostand$http_host. In most cases, you will want to set the "Host" header to the $host variable. It is the most flexible and will usually provide the proxied servers with a "Host" header filled in as accurately as possible.
The X-Forwarded-Proto header gives the proxied server information about the schema of the original client request (whether it was an http or an https request).
The X-Real-IP is set to the IP address of the client so that the proxy can correctly make decisions or log based on this information.
We could move the proxy_set_header directives out to the server or http context, allowing it to be referenced in more than one location.
Without buffers, data is sent from the proxied server and immediately begins to be transmitted to the client. If the clients are assumed to be fast, buffering can be turned off in order to get the data to the client as soon as possible. With buffers, the Nginx proxy will temporarily store the backend's response and then feed this data to the client. If the client is slow, this allows the Nginx server to close the connection to the backend sooner. It can then handle distributing the data to the client at whatever pace is possible.
More details in this article: Understanding nginx http proxying load balancing buffering and caching
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365
https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl
nginx -t
service nginx restart
Round-robin by default.
SQLite isn't a network database, so it doesn't have any network connection ability built into it. Recommended to use something else.
apt install mysql-server mysql-client libmysqlclient-dev python-dev
pip install mysql-python
easy_install mysql-python
Create a database named codebench and a database named codebench_test
on the newly installed MySQL instance. This can be achieved using
echo "CREATE DATABASE codebench character set UTF8 collate utf8_bin;CREATE DATABASE codebench_test set UTF8 collate utf8_bin;" | mysql -uroot -p<root MySQL password>
Grant remote access permission:
- Change mysql config
Start with mysql config file:
/etc/mysql/my.cnf
For me, it contains:
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
And the relevant line is in
/etc/mysql/mysql.conf.d/mysqld.cnf
Comment out this line by adding # in front:
bind-address = 127.0.0.1
- Restart mysql server.
service mysql reload
Different operating systems would need different commands, read more here.
- Change GRANT privilege
By default, mysql username and password you are using is allowed to access mysql-server locally. So need to update privilege.
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '<root MySQL password>' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
You can also specify a separate USERNAME & PASSWORD for remote access.
You can check final outcome by:
SELECT * from information_schema.user_privileges where grantee like "'root'%";
-
Test Connection, from the remote server terminal/command-line:
mysql -h HOST -u USERNAME -p
Read more here, also has examples on "Revoke Access".
May need to strict up those settings to implement a better security practice. Read more on Django security.
SELECT SCHEMA_NAME 'database', default_character_set_name 'charset', DEFAULT_COLLATION_NAME 'collation' FROM information_schema.SCHEMATA;
Django docs for database setup.
If you are moving from SQLite to MySQL, it's recommended to backup the data,
then delete all migration files (except the __init__.py), so you can makemigrations
from a clean state before try to migrate data to the new MySQL database.
python manage.py makemigrations
python manage.py migrate
You should only do it once on the database server, assuming you only have one database server.
Similar to above, comment out the bind 127.0.0.1 line in:
/etc/redis/redis.conf
TODO: Again security issues, need careful tuning on production server. Such as, adding an AUTH password in Redis and configuring your firewall (e.g. iptables) to block unauthorized clients. Read more on Stackoverflow, and this blog.
Restart the service
service redis-server restart
Test on remote server:
redis-cli -h [db server ip] ping
The sticky cookie method. With this method, NGINX adds a session cookie to the first response from the upstream group and identifies the server which has sent the response. When a client issues next request, it will contain the cookie value and NGINX will route the request to the same upstream server:
sticky cookie srv_id expires=1h;
The srv_id parameter sets the name of the cookie which will be set or inspected.
The optional expires parameter sets the time for the browser to keep the cookie.
The optional domain parameter defines a domain for which the cookie is set. The optional path parameter defines the path for which the cookie is set.
Sticky upstream is a third party module, check the list of third party Nginx modules here
sudo su -l aptitude update && aptitude dist-upgrade aptitude install build-essential software-properties-common
Remote desktop Edit /etc/xrdp/xrdp.ini
- make executing shell commands over SSH easy and Pythonic
- automate interactions with remote servers