01_setup.md

Ad hoc setup for a Swoop ML experimentation machine

Run the following:

curl 'https://gist.githubusercontent.com/ssimeonov/2319ecb00d825d6f5c78/raw/2bf43b3c5b766b9ce16f647fadbd7b423234f210/aws_ml_setup.sh' | bash -v

If the script exits without an error right after installing some packages, run it again.

If the script exits after aws configure, run aws configure yourself.

To sync view-click data from S3, do something like:

cd ~/spx/data
aws s3 sync s3://swoop-events-processed/view-clicks/ view-clicks/

aws_ml_setup.sh

#!/usr/bin/env bash
sudo apt-get update
sudo apt-get install git flex bison libtool make autoconf openssl libonig2 libonig-dev libssl-dev parallel tree

curl 'https://bootstrap.pypa.io/get-pip.py' | sudo python
sudo pip install --upgrade awscli

curl 'https://gist.githubusercontent.com/ssimeonov/2319ecb00d825d6f5c78/raw/6a7a18edb24a5556d578efaed3499b928bab6927/bash_profile' >> ~/.bash_profile
curl https://raw.githubusercontent.com/fesplugas/rbenv-installer/master/bin/rbenv-installer | bash
source ~/.bash_profile
rbenv install 2.1.5
rbenv global 2.1.5

mkdir ~/spx
cd ~/spx

wget http://www.geocities.jp/kosako3/oniguruma/archive/onig-5.9.6.tar.gz
tar zxvf onig-5.9.6.tar.gz
cd onig-5.9.6/
./configure
make
sudo make install

cd ~/spx

git clone https://github.com/stedolan/jq.git
cd jq
git checkout 902aa39
autoreconf -i
./configure --disable-maintainer-mode
make
sudo make install

sudo ln -s /usr/bin/sort /usr/bin/gsort
sudo ln -s /bin/zcat /bin/gzcat
sudo ln -s /bin/shuf /bin/gshuf

sudo mkdir /mnt/data
sudo chown ubuntu /mnt/data

cd ~/spx
ln -s /mnt/data data

cd ~/spx
mkdir data/by-request
mkdir data/view-clicks

cd data

cd ~/spx
git clone https://github.com/Shopximity/swoop-ml-nlp.git

aws configure
aws s3 ls s3://swoop-events-processed/grouped-by-request-id/2015/05/19/

bash_profile

export RBENV_ROOT="${HOME}/.rbenv"

if [ -d "${RBENV_ROOT}" ]; then
  export PATH="${RBENV_ROOT}/bin:${PATH}"
  eval "$(rbenv init -)"
fi

SPX_DEV_ROOT="${HOME}/spx"