Skip to content

Instantly share code, notes, and snippets.

@almet

almet/marketplace_installation.rst

Forked from tarekziade/marketplace_installation.rst
Created October 2, 2012 11:07
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save almet/3818214 to your computer and use it in GitHub Desktop.
Save almet/3818214 to your computer and use it in GitHub Desktop.
Download ZIP
Installing Marketplace in Ubuntu 12.04
Raw
marketplace_installation.rst

This is a condensed step-by-step guide on how to install Marketplace with all its co-servers (redis, ES, Rabittmq, MySQL etc)

More on Marketplace/Zamboni: http://zamboni.readthedocs.org/

Prerequisites

Ubuntu setup

  • when you install Ubuntu, use the default settings and create a user marketplace, password marketplace (or wathever you want - just adapt the tutorial)

  • once you are logged in your fresh ubuntu, run the following:

    sudo add-apt-repository ppa:fkrull/deadsnakes  
sudo apt-get update
sudo aptitude install python python2.6 python2.6-dev python-virtualenv libxml2-dev libxslt1-dev  libmysqlclient-dev libmemcached-dev libssl-dev swig git mysql-server  mysql-client subversion g++ make nginx rabbitmq-server redis-server openjdk-7-jre libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev ffmpeg

    Ubuntu will ask you for a root password for MySQL, use root (or wathever you want - just adapt the tutorial)

This going to last for ages if you live in the country like me.

Last but not least, edit the /etc/network/interfaces file in your fresh Ubuntu, so we define a static ip address to the second adapter add this:

auth eth1
iface eth1 inet static
    address 33.33.33.24
    netmask 255.255.255.0
    gateway 33.33.33.1

Restart the network and make sure eth1 show up with the right IP:

sudo /etc/init.d/networking restart
ifconfig

Try to ping the VBOx from your host:

ping 33.33.33.24

You should be ablt to reach it, and the VBox should be able to reach the net through the host.

Lessc & Uglify

Lessc is needed by Marketplace to create all the css. Ubuntu has a node-less package but I had issues with it, so I did a manual installation of node, nmp and lessc:

cd /tmp
git clone https://github.com/joyent/node.git
cd node && ./configure && make
sudo make install
cd ~
sudo curl -s  https://npmjs.org/install.sh | sh
sudo npm install less -g
sudo npm install uglify-js -g

ElasticSearch

Go get the latest Elastic Search debian image on their website. At the time of writting, this is 0.19.10.

wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.10.deb
dpkg -i elasticsearch-*.deb

Next, edit /etc/elasticsearch/config/elasticsearch.yml and add:

cluster:
  name: wooyeah

index:
  analysis:
    analyzer:
      standardPlusWordDelimiter:
        type: custom
        tokenizer: standard
        filter: [standard, wordDelim, lowercase, stop, dict]
    filter:
      wordDelim:
        type: word_delimiter
        preserve_original: true
      dict:
        type: dictionary_decompounder
        word_list_path: wordlist.txt

Then go into the config dir and download the wordlist:

cd /etc/elasticsearch/config
wget https://bug674810.bugzilla.mozilla.org/attachment.cgi?id=549013 -O wordlist.txt

Last, restart it:

sudo service elasticsearch restart.

RabbitMQ

Marketplace uses RabbitMQ through Celery to defer some tasks. It was installed previously. Just run it and add a zamboni user:

sudo rabbitmq-server -detached
sudo rabbitmqctl add_user zamboni zamboni
sudo rabbitmqctl add_vhost zamboni
sudo rabbitmqctl set_permissions -p zamboni zamboni ".*" ".*" ".*"

MySQL

MySQL should be already running. Let's create a marketplace user and DB:

mysql -uroot -proot

$(mysql shell) CREATE USER 'marketplace'@'localhost' IDENTIFIED BY 'marketplace';
$(mysql shell) CREATE DATABASE marketplace;
$(mysql shell) GRANT ALL PRIVILEGES ON marketplace.* to marketplace@localhost ;
$(mysql shell) exit

Zamboni

Let's install Zamboni now, which is the application that powers Marketplace:

cd ~
git clone --recursive git://github.com/mozilla/zamboni.git
cd zamboni
svn co http://svn.mozilla.org/addons/trunk/site/app/locale locale
curl -s https://raw.github.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | $SHELL
source ~/.venvburrito/startup.sh
mkvirtualenv marketplace -p python2.6

What I did here is cloned Zamboni and all its submodules, and created a virtualenv for it. For now on if you run Python, it will be the virtualenv one. If you exit it, you can just run:

workon marketplace

Now set up the pip download cache by creating ~/.pip/pip.conf with :

[global]
download-cache = /home/marketplace/.pip/downloads

That will reuse already downloaded files everytime you run Pip. This will speed things up for the next runs.

Then:

pip install -r requirements/dev.txt requirements/compiled.txt

Note

You may encounter some problems here, because M2Crypto doesn't play well with the SSLv2 symbol. The way to solve this is explained here: https://gist.github.com/3818341

If you go this way, be sure to remove the dependency to M2Crypto in the requirements/compiled.txt file.

Also, if you are running an x64 system, you may encounter some problems with PIL jpeg support, because it doesn't look for the .so files at the good place. Here is one way to fix it:

sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
# reinstall PIL
pip install -I PIL

Now let's configure the app settings:

cd ~/zamboni
cp docs/settings/settings_local.dev.py settings_local_mkt.py
ln -s settings_local_mkt.py settings_local.py

Now let's edit the file. You want to add those:

LESS_LIVE_REFRESH = True
LESS_PREPROCESS = True
SERVE_TMP_PATH = True 
RUN_ES_TESTS = True
LESS_BIN = 'lessc'
UGLIFY_BIN = 'uglifyjs'
SITE_URL_OVERRIDE = SITE_URL = STATIC_URL = 'http://addons.mozilla.dev'

Then change the Databse configuration so it looks like this:

DATABASES = {
    'default': {
        'NAME': 'marketplace',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'marketplace',
        'PASSWORD': 'marketplace',
        'OPTIONS':  {'init_command': 'SET storage_engine=InnoDB'},
        'TEST_CHARSET': 'utf8',
        'TEST_COLLATION': 'utf8_general_ci',
    },
}

Here's an example of a full settings file: https://gist.github.com/3818180

Initializing the Databases

We want to do 3 things: - fill the My SQL DB with the initial structure - compress the assets - get Elastic Search ready

For the DB SQL, run this command:

python manage.py --settings=settings_local_mkt install_landfill
export DJANGO_SETTINGS_MODULE='settings_local_mkt'
schematic migrations

I had several issues here. First I had to run the migration #260 manually. Second I had issues in django-mozilla-product-details. For the latter, have a look at https://github.com/fwenzel/django-mozilla-product-details/issues/27 . Applying that fix worked for me.

If you need to run migrations manually, you can do so with this command:

cat migrations/<migration_number>*.sql | mysql -uroot -proot marketplace

The next step is to compress the assets:

python manage.py --settings=settings_local_mkt compress_assets

That runs lessc and uglifyjs.

Last, let's prepare the Elastic Search DB:

python manage.py cron reindex_addons

Setting up the Host

We're almost done !

Since Marketplace uses BrowserId, we need to trick the Persona server. Add this in your /etc/hosts file (not in the Virtualbox guest!)

33.33.33.24  addons.mozilla.dev

Last, let's configure Nginx within the VM so it binds the port 80 and proxies all the requests to Django. I just changed the file at /etc/nginx/site-enabled/default, so the location / section looks like this:

location / {
  proxy_pass http://0.0.0.0:8000:
  proxy_set_header Host $host;
}

restart nginx:

sudo /etc/init.d/nginx restart

And start Django:

cd ~/zamboni
python manage.py runserver --settings=settings_local_mkt 0.0.0.0:8000

Then visit http://addons.mozilla.dev and try to log in.

Run the tests

python manage.py test --settings=settings_local_mkt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment