Skip to content

Instantly share code, notes, and snippets.

@Valve
Last active February 4, 2017 20:32
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save Valve/1ca13fc29ab8cb58b23d to your computer and use it in GitHub Desktop.
Save Valve/1ca13fc29ab8cb58b23d to your computer and use it in GitHub Desktop.
Sixpack, Gunicorn, Nginx, Supervisord, Ubuntu setup

1. Create a new EC2 instance, add TCP ports 22, 80 and 8080 to the security group

2. I did everything on Ubuntu 14.04.2 server 64bit
3. Log in
sudo apt-get update
sudo apt-get upgrade

redis

sudo apt-get install redis-server

sixpack deps

sudo apt-get install python-dev python-pip python-software-properties

generate sixpackweb secret key:

openssl rand -base64 32 | sed "s/[+=\/:]//g"

sixpack config, place it in /etc/sixpack/sixpack.conf

redis_port: 6379                        # Redis port
redis_host: localhost                   # Redis host
redis_prefix: sixpack                   # all Redis keys will be prefixed with this
redis_db: 1                             # DB number in redis

# The regex to match for robots
robot_regex: $^|trivial|facebook|MetaURI|butterfly|google|amazon|goldfire|sleuth|xenu|msnbot|bing|SiteUptime|Slurp|WordPress|ZIBB|ZyBorg|pingdom|bot|yahoo|slurp|java|fetch|spider|url|crawl|oneriot|abby|commentreader|twiceler
# ignored_ip_addresses: []                # List of IP
#
# asset_path: gen                         # Path for compressed assets to live. This path is RELATIVE to sixpack/static
secret_key: <secretkey>                   # secret key, required for sixpack-web to run

install sixpack

sudo pip install sixpack

install gunicorn

sudo pip install gunicorn

install supervisor

sudo apt-get install supervisor

create directory for sixpack logs

sudo mkdir /var/log/sixpack

edit sixpack config for supervisor

sudo vim /etc/supervisor/conf.d/sixpack.conf

[program:sixpack]
command=gunicorn sixpack.server:start --bind unix:/tmp/sixpack.sock -w 5 --access-logfile /var/log/sixpack/gunicorn-access.log
autostart=true
autorestart=true
stdout_logfile = /var/log/sixpack/supervisor-sixpack.log              ; Where to write log messages
redirect_stderr = true                                                ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8                       ; Set UTF-8 as default encoding

edit sixpackweb config for supervisor

sudo vim /etc/supervisor/conf.d/sixpackweb.conf

[program:sixpackweb]
command=gunicorn sixpack.web:start --bind unix:/tmp/sixpackweb.sock -w 2 --access-logfile /var/log/sixpack/sixpackweb-gunicorn-access.log
autostart=true
autorestart=true
stdout_logfile = /var/log/sixpack/supervisor-sixpackweb.log           ; Where to write log messages
redirect_stderr = true                                                ; Save stderr in the same log
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8                       ; Set UTF-8 as default encoding

read new sixpack configs in supervisorctl and start the servers

sudo supervisorctl
> reread
> update
> start all

install latest stable nginx

sudo add-apt-repository ppa:nginx/stable
sudo apt-get update
sudo apt-get install nginx

test the nginx installation

sudo service nginx start

find number of cpus/cores on your machine

sudo lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Stepping:              4
CPU MHz:               2500.060
BogoMIPS:              5000.12
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0

replace the default nginx configuration (located in /etc/nginx/nginx.conf) with this one:

user www-data;
# make it equal to number of cores * number of CPUs
worker_processes 4;
pid /run/nginx.pid;

events {
        worker_connections 1024;
}

http {
        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        include /etc/nginx/sites-enabled/*;
}

test new configuration

sudo nginx -t

install apache2 utils

sudo apt-get install apache2-utils

generate new passwords

openssl rand -base64 24 | sed "s/[+=\/:]//g"

sudo htpasswd -c /etc/nginx/.htpasswd sixpack sudo htpasswd /etc/nginx/.htpasswd sixpackweb

going to nginx installation dir

cd /etc/nginx

create new virtual host configuration for sixpack server

sudo vim sites-available/sixpack

upstream sixpack {
        # fail_timeout=0 means we always retry an upstream even if it failed
        # to return a good HTTP response (in case the Unicorn master nukes a
        # single worker for timing out).
        server unix:/tmp/sixpack.sock fail_timeout=0;
}
server {
        listen 80 default;
        location / {
                auth_basic "Restricted";
                auth_basic_user_file /etc/nginx/.htpasswd;
                proxy_pass http://sixpack;
        }
}

create new virtual host configuration for sixpack server

sudo vim sites-available/sixpackweb

upstream sixpackweb {
        # fail_timeout=0 means we always retry an upstream even if it failed
        # to return a good HTTP response (in case the Unicorn master nukes a
        # single worker for timing out).
        server unix:/tmp/sixpackweb.sock fail_timeout=0;
}
server {
        listen 8080;
        location / {
                auth_basic "Restricted";
                auth_basic_user_file /etc/nginx/.htpasswd;
                proxy_pass http://sixpackweb;
        }
}

create 2 symbolic links to 'turn on' the sixpack and sixpack web

sudo ln -s /etc/nginx/sites-available/sixpack /etc/nginx/sites-enabled/sixpack
sudo ln -s /etc/nginx/sites-available/sixpackweb /etc/nginx/sites-enabled/sixpackweb

remove default nginx virtual host

sudo rm /etc/nginx/sites-enabled/default

reload nginx

sudo service nginx reload

done!

@nickveenhof
Copy link

Nginx is not a necessary component in this guide. For example, we use AWS ELB's as our load balancers.

@nickveenhof
Copy link

If there is interest, we have cloudformation templates for setting up sixpack in a very similar way. It would be even easier if we could package sixpack in a docker image so that we could host it in AWS ECS service. I think this guide should become either all AWS components (such as elasticache for Redis) or fully independent so that people can make their own choice. Thanks for sharing :)

@Valve
Copy link
Author

Valve commented Apr 27, 2015

@nickveenhof Agree on all. I used nginx because I know it and we needed basic auth on top. Does sixpack work with elasticache instead of Redis?

@nickveenhof
Copy link

Elasticache is basically hosted redis - works very well with this. Basic Auth should probably be a separate layer. Ideally sixpack runs in AWS ECS in a docker image and you have a nginx docker image that provides the authentication, also running in Docker in ECS. This allows you to scale infintely using autoscaling groups and auto-recover without touching sixpack code. Connecting it to Redis in ElastiCache gives you the benefit of not maintaining a cluster of redis nodes. Only disadvantage here is that is works in a primary/replica setup so that you need to make sure you have enough backups in place to recover when needed.

@Valve
Copy link
Author

Valve commented Apr 28, 2015

Thanks! Need to get up-to-speed with ECS, never tried them

@christaggart
Copy link

@russelltaylor05
Copy link

russelltaylor05 commented May 5, 2016

@Valve This worked great for me. I'm up and running so thanks for sharing. One quick question for you though.

I need to serve things up over HTTPS and I'm hoping I can piggyback on top of Cloudflare's SSL instead of setting up my own certs. I've done this for other sites, but I'm not quite sure how to go about it with this configuration. I'm new to python and haven't really used Nginx all that much.

I've opened up port 443 on my EC2 box, an now I'm assuming i need to configure Nginx or the Virtual Host to serve up these requests. Anything you can do to point me in the right direction would be greatly appreciated.

@koAlech
Copy link

koAlech commented May 23, 2016

Hi everybody,
We created a complete dockerized sixpack environment solution behind an HTTPS nginx proxy.
Just shared a bunch of scripts using our Docker Hub repository for get a sixpack server up and running in practically minutes.

Please check it out @ https://github.com/baloota/sixpack-complete
We'd really appreciate your feedback and pull-requests :)

Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment