Skip to content

Instantly share code, notes, and snippets.

@tarekziade

tarekziade/marketplace_installation.rst

Created September 24, 2012 10:09
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save tarekziade/3775297 to your computer and use it in GitHub Desktop.
Save tarekziade/3775297 to your computer and use it in GitHub Desktop.
Download ZIP
Installing Marketplace in Ubuntu 12.04
Raw
marketplace_installation.rst

This is a condensed step-by-step guide on how to install Marketplace with all its co-servers (redis, ES, Rabittmq, MySQL etc)

More on Marketplace/Zamboni: http://zamboni.readthedocs.org/

Prerequisites

Ubuntu setup

  • when you install Ubuntu, use the default settings and create a user marketplace, password marketplace (or wathever you want - just adapt the tutorial)

  • once you are logged in your fresh ubuntu, run the following:

    sudo apt-get install python-software-properties
sudo add-apt-repository ppa:fkrull/deadsnakes  
sudo apt-get update
sudo aptitude install python python2.6 python2.6-dev python-virtualenv libxml2-dev libxslt1-dev  libmysqlclient-dev libmemcached-dev libssl-dev swig git mysql-server  mysql-client subversion g++ make nginx rabbitmq-server redis-server openjdk-7-jre libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev ffmpeg

    Ubuntu will ask you for a root password for MySQL, use root (or wathever you want - just adapt the tutorial)

This going to last for ages if you live in the country like me.

Last but not least, edit the /etc/network/interfaces file in your fresh Ubuntu, so we define a static ip address to the second adapter add this:

auth eth1
iface eth1 inet static
    address 33.33.33.24
    netmask 255.255.255.0
    gateway 33.33.33.1

Restart the network and make sure eth1 show up with the right IP:

sudo /etc/init.d/networking restart
ifconfig

Try to ping the VBOx from your host:

ping 33.33.33.24

You should be ablt to reach it, and the VBox should be able to reach the net through the host.

Lessc & Uglify

Lessc is needed by Marketplace to create all the css. Ubuntu has a node-less package but I had issues with it, so I did a manual installation of node, nmp and lessc:

cd /tmp
git clone https://github.com/joyent/node.git
cd node && ./configure && make
sudo make install
cd ~
sudo curl -s  https://npmjs.org/install.sh | sh
sudo npm install less -g
sudo npm install uglify-js -g

ElasticSearch

Manual install here, like lessc :/ -- taken from https://gist.github.com/2026107 :

cd /tmp
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.9.tar.gz -O elasticsearch.tar.gz
tar -xf elasticsearch.tar.gz
rm elasticsearch.tar.gz
sudo mv elasticsearch-* elasticsearch
sudo mv elasticsearch /usr/local/share
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service /usr/local/share/elasticsearch/bin/
rm -Rf *servicewrapper*
sudo /usr/local/share/elasticsearch/bin/service/elasticsearch install
sudo ln -s `readlink -f /usr/local/share/elasticsearch/bin/service/elasticsearch` /usr/local/bin/rcelasticsearch
sudo service elasticsearch start

Next, edit /usr/local/share/elasticsearch/config/elasticsearch.yml and add:

cluster:
  name: wooyeah

index:
  analysis:
    analyzer:
      standardPlusWordDelimiter:
        type: custom
        tokenizer: standard
        filter: [standard, wordDelim, lowercase, stop, dict]
    filter:
      wordDelim:
        type: word_delimiter
        preserve_original: true
      dict:
        type: dictionary_decompounder
        word_list_path: wordlist.txt

Then go into the config dir and download the wordlist:

cd /usr/local/share/elasticsearch/config
wget https://bug674810.bugzilla.mozilla.org/attachment.cgi?id=549013 -O wordlist.txt

Last, restart it:

sudo /etc/init.d/elasticsearch restart.

RabbitMQ

Marketplace uses RabbitMQ through Celery to defer some tasks. It was installed previously. Just run it and add a zamboni user:

sudo rabbitmq-server -detached
sudo rabbitmqctl add_user zamboni zamboni
sudo rabbitmqctl add_vhost zamboni
sudo rabbitmqctl set_permissions -p zamboni zamboni ".*" ".*" ".*"

MySQL

MySQL should be already running. Let's create a marketplace user and DB:

mysqladmin -uroot -proot

$(mysql shell) CREATE USER 'marketplace'@'localhost' IDENTIFIED BY 'marketplace';
$(mysql shell) CREATE DATABASE marketplace;
$(mysql shell) GRANT ALL PRIVILEGES ON marketplace.* to marketplace@localhost ;
$(mysql shell) exit

Zamboni

Let's install Zamboni now, which is the application that powers Marketplace:

cd ~
git clone --recursive git://github.com/mozilla/zamboni.git
cd zamboni
svn co http://svn.mozilla.org/addons/trunk/site/app/locale locale
curl -s https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | $SHELL
source ~/.venvburrito/startup.sh
mkvirtualenv marketplace -p python2.6

What I did here is cloned Zamboni and all its submodules, and created a virtualenv for it. For now on if you run Python, it will be the virtualenv one. If you exit it, you can just run:

workon marketplace

Now set up the pip download cache by creating ~/.pip/pip.conf with :

[global]
download-cache = /home/marketplace/.pip/downloads

That will reuse already downloaded files everytime you run Pip. This will speed things up for the next runs.

Then:

pip install -r requirements/dev.txt

Now let's configure the app settings:

cd ~/zamboni
cp docs/settings/settings_local.dev.py settings_local_mkt.py
ln -s settings_local_mkt.py settings_local.py

Now let's edit the file. You want to add those:

LESS_LIVE_REFRESH = True
LESS_PREPROCESS = True
SERVE_TMP_PATH = True 
RUN_ES_TESTS = True
LESS_BIN = 'lessc'
UGLIFY_BIN = 'uglifyjs'
SITE_URL_OVERRIDE = SITE_URL = STATIC_URL = 'http://addons.mozilla.dev'

Then change the Databse configuration so it looks like this:

DATABASES = {
    'default': {
        'NAME': 'marketplace',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'marketplace',
        'PASSWORD': 'marketplace',
        'OPTIONS':  {'init_command': 'SET storage_engine=InnoDB'},
        'TEST_CHARSET': 'utf8',
        'TEST_COLLATION': 'utf8_general_ci',
    },
}

Here's an example of a full settings file: https://gist.github.com/3818180

Initializing the Databases

We want to do 3 things: - fill the My SQL DB with the initial structure - compress the assets - get Elastic Search ready

For the DB SQL, run this command:

python manage.py --settings=settings_local_mkt install_landfill
schematic migrations

I had several issues here. First I had to run the migration #260 manually. Second I had issues in django-mozilla-product-details. For the latter, have a look at https://github.com/fwenzel/django-mozilla-product-details/issues/27 . Applying that fix worked for me.

The next step is to compress the assets:

python manage.py --settings=settings_local_mkt compress_assets

That runs lessc and uglifyjs.

Last, let's prepare the Elastic Search DB:

python manage.py cron reindex_addons

Setting up the Host

We're almost done !

Since Marketplace uses BrowserId, we need to trick the Persona server. Add this in your /etc/hosts file (not in the Virtualbox guest!)

33.33.33.24  addons.mozilla.dev

Last, let's configure Nginx within the VM so it binds the port 80 and proxies all the requests to Django. I just changed the file at /etc/nginx/site-enabled/default, so the location / section looks like this:

location / {
  proxy_pass http://0.0.0.0:8000:
  proxy_set_header Host $host;
}

restart nginx:

sudo /etc/init.d/nginx restart

And start Django:

cd ~/zamboni
python manage.py runserver --settings=settings_local_mkt 0.0.0.0:8000

Then visit http://addons.mozilla.dev and try to log in.

Run the tests

Install nose-progressive:

pip install nose-progressive

Then run the tests :

python manage.py test --noinput -s --logging-clear-handlers --with-progressive --settings=settings_local_mkt mkt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment