Azure setup

language-abstractions-azure-setup.md

Creating a new snapshot at the command line.

Create a resource group.

az vm group -n NAME -l LOCATION

Create a machine.

az vm create --size Standard_E64_v3 -n ec2_base_image_0 --location centralus --resource-group ec2_resource_group_0 --admin-username azureuser --image Canonical:UbuntuServer:16.04-LTS:latest --generate-ssh-keys

Set the ulimit for file reading.

ulimit -n 500000

TODO pipe yes and force script

Copy the branch

git clone https://github.com/ellisk42/ec.git -b language_compression ; cd ec; mkdir jobs; mkdir data/logo; mkdir data/clevr; mkdir data/re2;

Install Python

sudo apt-get update; sudo apt-get install software-properties-common; sudo add-apt-repository ppa:deadsnakes/ppa; sudo apt-get update; sudo apt-get install python3.7;

sudo apt install python3.7-dev; sudo apt install python3-pip; python3.7 -m pip install pip; pip install --upgrade setuptools; pip install -r language_requirements.txt;

Download NLTK

nltk.download('punkt')

Optional

from torchnlp.word_to_vector import FastText; vectors = FastText()

Copy local data

mkdir data/clevr;

From local repo.

scp -r data/clevr/* zyzzyva@104.40.89.130:ec/data/clevr/ scp -r data/logo/* zyzzyva@104.40.89.130:ec/data/logo/ scp -r data/re2/* zyzzyva@104.40.89.130:ec/data/re2/

scp -r data/clevr/* zyzzyva@52.168.179.152:ec/data/clevr/ scp -r data/logo/* zyzzyva@52.168.179.152:ec/data/logo/ scp -r data/re2/* zyzzyva@52.168.179.1523:ec/data/re2/

Download Moses

wget http://www.statmt.org/moses/RELEASE-4.0/binaries/ubuntu-16.04.tgz ; tar -xvf ubuntu-16.04.tgz; mv ubuntu-16.04/ moses_compiled ; rm -rf ubuntu-16.04.tgz;

Install OCaml

sudo apt install ocaml ; sudo apt install opam ; opam init; opam update; opam switch 4.06.1+flambda ; eval opam config env; opam depext ppx_jane core re2 yojson vg cairo2 camlimages menhir ocaml-protoc zmq ; opam install ppx_jane core re2 yojson vg cairo2 camlimages menhir ocaml-protoc zmq ; make clean; make

TEST and CHANGE BACK TO 64G before starting again!

Updated version: we now use singularity for most of our installation.

Copy the branch

git clone https://github.com/ellisk42/ec.git -b master_language ; cd ec; mkdir ../ec_language_logs

Download Moses

wget http://www.statmt.org/moses/RELEASE-4.0/binaries/ubuntu-16.04.tgz ; tar -xvf ubuntu-16.04.tgz; mv ubuntu-16.04/ moses_compiled ; rm -rf ubuntu-16.04.tgz;

Install OCaml

sudo apt-get update; sudo apt install ocaml ; sudo apt install opam ; opam init; opam update; opam switch 4.06.1+flambda ; eval opam config env; opam depext ppx_jane core re2 yojson vg cairo2 camlimages menhir ocaml-protoc zmq ; opam install ppx_jane core re2 yojson vg cairo2 camlimages menhir ocaml-protoc zmq ; make clean; make

Install Singularity and grab the dev container

https://sylabs.io/guides/3.4/user-guide/installation.html (3.4.1) - we use the install from source Download the dev container -- scp zyzzyva@openmind7.mit.edu:/om2/user/zyzzyva/dev-container.img .

Grab the local data.

scp from a local directory.