Skip to content

Instantly share code, notes, and snippets.

@spladug
Forked from reddit/install-reddit.sh
Created April 15, 2011 17:59
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
  • Save spladug/922144 to your computer and use it in GitHub Desktop.
Save spladug/922144 to your computer and use it in GitHub Desktop.
#!/bin/bash -e
# Neil Williams, reddit
# configuration that doesn't change much
REDDIT_REPO=git://github.com/reddit/reddit.git
I18N_REPO=git://github.com/reddit/reddit-i18n.git
APTITUDE_OPTIONS="-y" # limit bandwidth: -o Acquire::http::Dl-Limit=100"
# don't blunder on if an error occurs
set -e
# validate some assumptions of the script
if [ $(id -u) -ne 0 ]; then
echo "ERROR: Must be run with root privileges."
exit 1
fi
source /etc/lsb-release
if [ "$DISTRIB_ID" != "Ubuntu" ]; then
echo "ERROR: Unknown distribution $DISTRIB_ID. Only Ubuntu is supported."
exit 1
fi
if [ "$DISTRIB_RELEASE" != "11.04" ]; then
echo "ERROR: Only Ubuntu 11.04 is supported."
exit 1
fi
# get configured
echo "Welcome to the reddit install script!"
REDDIT_USER=${SUDO_USER:-reddit}
read -ei $REDDIT_USER -p "What user should reddit run as? " REDDIT_USER
REDDIT_HOME=/home/$REDDIT_USER
read -ei $REDDIT_HOME -p "What directory should it be installed in? " REDDIT_HOME
echo "Beginning installation. This may take a while..."
echo
# create the user if non-existent
if ! id $REDDIT_USER > /dev/null; then
adduser --system $REDDIT_USER
fi
# add some external ppas for packages
DEBIAN_FRONTEND=noninteractive aptitude install $APTITUDE_OPTIONS python-software-properties
apt-add-repository ppa:jason-alioth/reddit
# pin the ppa
cat <<HERE > /etc/apt/preferences.d/reddit
Package: *
Pin: release o=LP-PPA-jason-alioth-reddit
Pin-Priority: 600
HERE
# grab the new ppas' package listings
aptitude update
# install prerequisites
DEBIAN_FRONTEND=noninteractive aptitude install $APTITUDE_OPTIONS python-dev python-setuptools python-imaging python-pycaptcha python-mako python-nose python-decorator python-formencode python-pastescript python-beaker python-webhelpers python-amqplib python-pylibmc python-pycountry python-psycopg2 python-cssutils python-beautifulsoup python-sqlalchemy cython python-pybabel python-tz python-boto python-lxml python-pylons python-pycassa python-recaptcha gettext make optipng uwsgi uwsgi-core uwsgi-plugin-python nginx git-core python-profiler memcached postgresql postgresql-client curl daemontools daemontools-run rabbitmq-server cassandra
# grab the source
if [ ! -d $REDDIT_HOME ]; then
mkdir $REDDIT_HOME
chown $REDDIT_USER $REDDIT_HOME
fi
cd $REDDIT_HOME
if [ ! -d $REDDIT_HOME/reddit ]; then
sudo -u $REDDIT_USER git clone $REDDIT_REPO
fi
if [ ! -d $REDDIT_HOME/reddit-i18n ]; then
sudo -u $REDDIT_USER git clone $I18N_REPO
fi
# wait a bit to make sure all the servers come up
sleep 30
# configure cassandra
if ! echo | cassandra-cli -h localhost -k reddit > /dev/null 2>&1; then
echo "create keyspace reddit;" | cassandra-cli -h localhost -B
fi
echo "create column family permacache with column_type = 'Standard' and comparator = 'BytesType';" | cassandra-cli -B -h localhost -k reddit || true
# set up postgres
IS_DATABASE_CREATED=$(sudo -u postgres psql -t -c "SELECT COUNT(1) FROM pg_catalog.pg_database WHERE datname = 'reddit';")
if [ $IS_DATABASE_CREATED -ne 1 ]; then
cat <<PGSCRIPT | sudo -u postgres psql
CREATE DATABASE reddit WITH ENCODING = 'utf8';
CREATE USER reddit WITH PASSWORD 'password';
PGSCRIPT
fi
sudo -u postgres psql reddit < $REDDIT_HOME/reddit/sql/functions.sql
# set up rabbitmq
if ! rabbitmqctl list_vhosts | egrep "^/$"
then
rabbitmqctl add_vhost /
fi
if ! rabbitmqctl list_users | egrep "^reddit"
then
rabbitmqctl add_user reddit reddit
fi
rabbitmqctl set_permissions -p / reddit ".*" ".*" ".*"
# run the reddit setup script
cd $REDDIT_HOME/reddit/r2
sudo -u $REDDIT_USER make pyx # generate the .c files from .pyx
sudo -u $REDDIT_USER python setup.py build
python setup.py develop
# run the i18n setup script
cd $REDDIT_HOME/reddit-i18n/
sudo -u $REDDIT_USER python setup.py build
python setup.py develop
sudo -u $REDDIT_USER make
# do the r2 build after languages are installed
cd $REDDIT_HOME/reddit/r2
sudo -u $REDDIT_USER make
# install the daemontools runscripts
cd $REDDIT_HOME/reddit/r2
if [ ! -L run.ini ]; then
sudo -u $REDDIT_USER ln -s example.ini run.ini
fi
ln -s $REDDIT_HOME/reddit/srv/* /etc/service/ || true
/sbin/start svscan || true
# set up uwsgi
cat >/etc/uwsgi/apps-available/reddit.ini <<UWSGI
[uwsgi]
; master / uwsgi protocol configuration
plugins = python
master = true
master-as-root = true
vacuum = true
limit-post = 512000
buffer-size = 8096
uid = $REDDIT_USER
chdir = $REDDIT_HOME/reddit/r2
socket = /tmp/reddit.sock
chmod-socket = 666
; worker configuration
lazy = true
max-requests = 10000
processes = 1
; app configuration
paste = config:$REDDIT_HOME/reddit/r2/run.ini
UWSGI
if [ ! -L /etc/uwsgi/apps-enabled/reddit.ini ]; then
ln -s /etc/uwsgi/apps-available/reddit.ini /etc/uwsgi/apps-enabled/reddit.ini
fi
/etc/init.d/uwsgi start
# set up nginx
cat >/etc/nginx/sites-available/reddit <<NGINX
include uwsgi_params;
uwsgi_param SCRIPT_NAME "";
server {
listen 8000;
location / {
uwsgi_pass unix:///tmp/reddit.sock;
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_min_length 100;
gzip_comp_level 4;
gzip_types text/plain text/css text/javascript text/xml application/json text/csv application/x-javascript application/xml application/xml+rss;
}
}
NGINX
if [ ! -L /etc/nginx/sites-enabled/reddit ]; then
ln -s /etc/nginx/sites-available/reddit /etc/nginx/sites-enabled/reddit
fi
/etc/init.d/nginx restart
# install the crontab
CRONTAB=$(mktemp)
crontab -u $REDDIT_USER -l > $CRONTAB || true
cat >>$CRONTAB <<CRON
# m h dom mon dow command
*/5 * * * * $REDDIT_HOME/reddit/scripts/rising.sh
*/4 * * * * $REDDIT_HOME/reddit/scripts/send_mail.sh
*/3 * * * * $REDDIT_HOME/reddit/scripts/broken_things.sh
1 * * * * $REDDIT_HOME/reddit/scripts/update_promos.sh
0 23 * * * $REDDIT_HOME/reddit/scripts/update_reddits.sh
30 23 * * * $REDDIT_HOME/reddit/scripts/update_sr_names.sh
CRON
crontab -u $REDDIT_USER $CRONTAB
rm $CRONTAB
# done!
cd $REDDIT_HOME
echo "Done installing reddit!"
@dweitz
Copy link

dweitz commented Jun 14, 2011

The link to the reddit version of Paste (line 132) doesn't work anymore. Any other place to download it?

@spladug
Copy link
Author

spladug commented Jun 14, 2011

The package hangs out here. That should've automatically been sourced by our setup.py. Did that not work?

@dweitz
Copy link

dweitz commented Jun 15, 2011

It was timing out for me before for that one file and giving me 404 errors when I tried to download from a browser. Seems to be working now.

@cheapsteak
Copy link

Ran into this error when I tried to install on a fresh ubuntu:
paster run standalone r2/lib/js.py -c "build_reddit_js()"
Traceback (most recent call last):
File "/usr/local/bin/paster", line 9, in
load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
File "/usr/local/lib/python2.6/dist-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 83, in run
command = commands[command_name].load()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1948, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/home/reddit/reddit/r2/r2/init.py", line 26, in
from r2.config.middleware import make_app
File "/home/reddit/reddit/r2/r2/config/init.py", line 22, in
from r2.lib.cache import sgm
File "/home/reddit/reddit/r2/r2/lib/cache.py", line 35, in
from r2.lib.utils import in_chunks, prefix_keys, trace
File "/home/reddit/reddit/r2/r2/lib/utils/init.py", line 22, in
from utils import *
File "/home/reddit/reddit/r2/r2/lib/utils/utils.py", line 36, in
from r2.lib.filters import _force_unicode
File "/home/reddit/reddit/r2/r2/lib/filters.py", line 29, in
from lxml.sax import saxify
ImportError: No module named lxml.sax
make: *** [r2/public/static/button.js] Error 1

had to manually install lxml for it to work, with
sudo apt-get install python-lxml

@cheapsteak
Copy link

The newest version of pycountry 0.14 which is installed by default doesn't seem to work with this script, threw this error
File '/usr/local/lib/python2.6/dist-packages/pycountry-0.14-py2.6.egg/pycountry/init.py', line 16 in
class Countries(pycountry.db.Database):
AttributeError: 'module' object has no attribute 'db'

installing 0.12.1 fixed it
easy_install "pycountry==0.12.1"

@Chiggins
Copy link

For this to still work in (at least Ubuntu 10.10), you should change Cassandra's package name to include a previous version, since getting Reddit up in version 0.8 with the current configs wont work. "cassandra=0.7.2-0ubuntu4" would work.

@artlogic
Copy link

pycountry has been updated and now works with the latest version

@rpicard
Copy link

rpicard commented Sep 28, 2011

I ran into some problems using this on Ubuntu 11.04. I could be doing something wrong but I kept getting an error saying that aptitude was not a recognized command. I changed all of the references from aptitude to apt-get and it seems to have solved the problem. I just installed Ubuntu today to start playing with Reddit. I think the problem is that it doesn't come with aptitude installed (and it did in previous versions?).

See this gist for the modified version.
https://gist.github.com/1247021

@Chiggins
Copy link

aptitude isn't installed by default in 11.04. I BELIEVE it's something of an upgrade to apt-get, but I can't fully confirm that.

@rpicard
Copy link

rpicard commented Sep 28, 2011

I figured it was something like that. I ran into a few more errors after I got passed that and I realized I was getting in a little over my head with this. I'm going to stick with simple PHP scripts for now.

@gordoncl
Copy link

Aptitude is not installed by default. I just did a fresh install of 11.04. But, you can easily install it by running:
apt-get update
apt-get install aptitude

@andrewtuplin
Copy link

I'm attempting to install on 11.1 (I removed the bit in the .sh that checks for version 11.04) and the install bails when it gets to the PPA:

You are about to add the following PPA to your system:
reddit
Packages for reddit
More info: https://launchpad.net/~jason-alioth/+archive/reddit

(everything is going fine until... )

Hit http://us.archive.ubuntu.com oneiric-backports/multiverse TranslationIndex
Hit http://us.archive.ubuntu.com oneiric-backports/restricted TranslationIndex
Err http://ppa.launchpad.net oneiric/main Sources
404 Not Found
Err http://ppa.launchpad.net oneiric/main i386 Packages
404 Not Found
Hit http://us.archive.ubuntu.com oneiric-backports/universe TranslationIndex
Hit http://us.archive.ubuntu.com oneiric/main Translation-en
Hit http://us.archive.ubuntu.com oneiric/multiverse Translation-en
Hit http://us.archive.ubuntu.com oneiric/restricted Translation-en
Hit http://us.archive.ubuntu.com oneiric/universe Translation-en
Hit http://us.archive.ubuntu.com oneiric-updates/main Translation-en
Hit http://us.archive.ubuntu.com oneiric-updates/multiverse Translation-en
Hit http://us.archive.ubuntu.com oneiric-updates/restricted Translation-en
Hit http://us.archive.ubuntu.com oneiric-updates/universe Translation-en
Hit http://us.archive.ubuntu.com oneiric-backports/main Translation-en
Hit http://us.archive.ubuntu.com oneiric-backports/multiverse Translation-en
Hit http://us.archive.ubuntu.com oneiric-backports/restricted Translation-en
Hit http://us.archive.ubuntu.com oneiric-backports/universe Translation-en
Ign http://ppa.launchpad.net oneiric/main Translation-en_US
Ign http://ppa.launchpad.net oneiric/main Translation-en

root@ubuntu:/home/andrew/Downloads/gist922144-db8e2286a66ef18459cc356869df6d17e7b8f66a#

Any ideas on what's causing the script to bail?

@spladug
Copy link
Author

spladug commented Nov 8, 2011

Yup, none of the packages in the PPA are compiled for Oneiric. There's a reason that version check is there :P

@andrewtuplin
Copy link

Silly Me.

After reinstalling the correct version of Ubuntu, I'm bailing on the same error as Cheapsteak:

ImportError: No module named lxml.sax
make: *** [r2/public/static/button.js] Error 1

I've manually installed lxml with "sudo apt-get install python-lxml" but the script still dies at this line.

@ttanner
Copy link

ttanner commented Dec 1, 2011

I've tried to follow the steps on Lucid (10.04.3 LTS) but it fails on
$ make pyx
Traceback (most recent call last):
File "r2/lib/js.py", line 8, in
from r2.lib.i18n import get_available_languages
ImportError: No module named r2.lib.i18n
Traceback (most recent call last):
File "r2/lib/js.py", line 8, in
from r2.lib.i18n import get_available_languages
ImportError: No module named r2.lib.i18n
cython r2/models/_builder.pyx

Error converting Pyrex file to C:

...
def _tree_filter(self, x):
return tree_sort_fn(x) < self.after._id

def _viewable_message(self, m):
    if (c.user_is_admin
        or getattr(m, "author_id", 0) == c.user._id
                 ^

/home/reddit/reddit/r2/r2/models/_builder.pyx:282:22: Call with wrong number of arguments (expected 2, got 3)

@spladug
Copy link
Author

spladug commented Dec 1, 2011

You need a newer version of Cython.

Note: we don't support Lucid any more and there're parts of the code that will not work on Python 2.6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment