H2O driverless AI installation instructions

README.md

Installation scripts for H2O driverless AI

Launch order

• install.sh (requires sudo)
  • need to exit and re-ssh into machine for "docker usage without sudo" to take effect
• for GPU, refer to init-gpu.sh
• dropbox.sh (optional)
• start.sh

dropbox.sh

# Note 1: At the step of installing dropbox, open link displayed in screen to link your dropbox account
# for convenience, link the driverless AI folders to a dropbox folder
# This script is not meant to be run automatically in one shot
# Instead, run it manually one step at a time, modifying steps as needed
#############################

# install requirements for dropbox headless for convenience to upload files into driverless AI
# https://www.dropbox.com/install-linux
sudo apt-get -qq -y install \
    python
sudo wget https://www.dropbox.com/download?dl=packages/dropbox.py -O /usr/bin/dropbox
sudo chmod +x /usr/bin/dropbox

# mount a volume from /dev/xvdf into /mnt/ec2vol
# and put the dropbox folder there
# to format a new volume: mkfs -t ext4 /dev/xvdf
sudo mkdir /mnt/ec2vol
sudo mount /dev/xvdf /mnt/ec2vol
sudo chown ubuntu:ubuntu /mnt/ec2vol -R

# This is where the prompt in Note 1 will show up
# move dropbox folder to /mnt/ec2vol
cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
~/.dropbox-dist/dropboxd # manually click the link, when done linking the account, press Ctrl+C to terminate

# selective sync
# http://rkulla.blogspot.com/2014/03/headless-dropbox.html
dropbox exclude add "AEE-Office"
dropbox exclude add "AEE-Stores"
dropbox exclude add "Akiki VHS"
dropbox exclude add "Lebanon2014_Jim"

# start again
dropbox start

#------------------------------------------------------------------------------------
# migrate an existing dropbox folder
# NOT SURE IF THIS WORKS. HAVE FAILED EVERYTIME SO FAR
watch dropbox status # wait till it starts syncing, then do dropbox stop
mv ~/Dropbox /mnt/ec2vol/Dropbox
ln -s /mnt/ec2vol/Dropbox ~/Dropbox

# At this step, make sure no files get "deleted" by dropbox unintentionally
# If it happens, log into the web interface, click "show deleted files", select them, and click "restore"
dropbox start 

#------------------------------------------------------------------------------------
# link the dropbox folder to the /var/lib/h2o-driverless-ai folder that will be mount into the docker image
mkdir ~/Dropbox/Shadi_Datasets/h2o-driverless-ai/{mkdir,log,license}
cd /var/lib
sudo mv driverless-ai-dir driverless-ai-dir.bkp
sudo ln -s ~/Dropbox/Shadi_Datasets/h2o-driverless-ai/ .
cd -

install.sh

# Script to install driverless AI from H2O
# Usage
#        wget https://gist.github.com/shadiakiki1986/71f49c4051b68a2a9d6e449b62d00a8e/raw/install.sh -O -|sudo bash
#
# Requirements
#        needs to be used with sudo to run
#        minimum free disk space 13 GB
#        ubuntu 16 or 17
#        A dropbox account for simplifying of uploading files into driverless AI's data folder
#        Adds the user "ubuntu" to the docker group to be able to run docker without sudo
#
# Note: permalink from https://gist.github.com/atenni/5604615
######################################################################################################################################################################

set -e
export DEBIAN_FRONTEND=noninteractive

# install docker
# prepend h2o's documentation with getting the docker's official gpg key
# https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#os-requirements
apt-get -qq -y remove docker docker-engine docker.io
apt-get -qq -y update
apt-get -qq -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
    
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88

# continue with h2o's documentation
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get -qq -y update
apt-get -qq -y install docker-ce
docker run hello-world

# add user to docker group
gpasswd -a ubuntu docker

# Install nvidia-docker. Required only if GPU is used
wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
dpkg -i /tmp/nvidia-docker*.deb
rm /tmp/nvidia-docker*.deb

# without this, the nvidia-docker run... below would fail
apt-get -qq -y install nvidia-modprobe

# download the driverless ai docker image (2GB)
wget -P /tmp https://s3-us-west-2.amazonaws.com/h2o-internal-release/docker/driverless-ai-docker-runtime-rel-0.8.2.gz
# load it in docker (slow command)
sleep 1 # is this necessary for the wget to move the file to /tmp ? Not sure
docker load < /tmp/driverless-ai-docker-runtime-rel*.gz # <<< does this only work with bash and not /bin/sh?
rm /tmp/driverless-ai-docker-runtime-rel*.gz

# prepare folders
mkdir -p /var/lib/h2o-driverless-ai/{data,log,license}

launch-h2o-ai.sh

#!/bin/bash
# launch h2o.ai locally
# mounts the data/log/license directories from /var/lib/h2o-driverless-ai
# publishes to port 3001

# cannot use this with nvidia-modprobe failing with no GPU
# set -e

ROOT=/var/lib/h2o-driverless-ai
RUNNER=nvidia-docker

# the below will exit with non-0 if no gpu
nvidia-modprobe
if [ $? -eq 1 ]; then
  echo "no gpu found"
  RUNNER=docker
fi

echo "launch on port 3001. Check http://ip:3001"
$RUNNER run \
  --rm \
  -u `id -u`:`id -g` \
  -p 3001:12345 \
  -v $ROOT/data:/data \
  -v $ROOT/log:/log \
  -v $ROOT/license:/license \
   opsh2oai/h2oai-runtime

scorers.sh

# This is an example script of requirements for running a scorer.zip file
# exported from h2o driverless-ai
# Assumes filename is scorer.zip
#############################################

# install python3.6 requirement
sudo add-apt-repository ppa:jonathonf/python-3.6
sudo apt-get update
sudo apt-get install python3.6 python3.6-dev

# install some openmp stuff, and fix a missing link
sudo apt-get install libomp5
sudo ln -s /usr/lib/x86_64-linux-gnu/libomp.so.5  /usr/lib/x86_64-linux-gnu/libomp.so

unzip scorer.zip # could use -d /path/to/another/folder
cd python_module

# instead of using the run_example.sh which uses virtualenv
# use pew, because pew is awesome
# https://github.com/berdario/pew
sudo pip3 install pew
pew new -r requirements.txt --python=python3.6 H2O_SCORER
python example.py

Thanks for sharing this. I found your scripts a bit more helpful than the h2o documentation!

You're welcome :)