How to run Jupyter Notebook on NYU HPC to run tensorflow

README.md

Usage

0. If you don't have a virtual environment, download create_venv.sh and do this in your working directory:

   source create_venv.sh

1. Run run-jupyter.sbatch and follow the instructions in NYU Prince Tutorial on how to run Jupyter Notebook:

   sbatch run-jupyter.sbatch

create_venv.sh

#!/bin/bash

# activate existed python3 module to get virtualenv
module load python3/intel/3.6.3

# create virtual environment with python3
virtualenv -p python3 .env

# activate virtual environment
source .env/bin/activate

# install tensorflow-gpu
pip install --upgrade pip
pip install tensorflow-gpu
pip install jupyter
# uncomment the following line if you have requirements.txt file
# pip install -r requirements.txt

# unload module
module unload python3/intel/3.6.3

run-jupyter.sbatch

#!/bin/bash

#SBATCH --job-name=jupyter
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16GB
#SBATCH --time=12:00:00

module purge
module load cuda/9.0.176 cudnn/9.0v7.0.5

source .env/bin/activate
port=$(shuf -i 6000-9999 -n 1)

/usr/bin/ssh -N -f -R $port:localhost:$port log-0
/usr/bin/ssh -N -f -R $port:localhost:$port log-1

cat<<EOF

Jupyter server is running on: $(hostname)
Job starts at: $(date)

Step 1 :

If you are working in NYU campus, please open an iTerm window, run command

ssh -L $port:localhost:$port $USER@prince.hpc.nyu.edu

If you are working off campus, you should already have ssh tunneling setup through HPC bastion host,
that you can directly login to prince with command

ssh $USER@prince

Please open an iTerm window, run command

ssh -L $port:localhost:$port $USER@prince

Step 2:

Keep the iTerm windows in the previouse step open. Now open browser, find the line with

The Jupyter Notebook is running at: $(hostname)

the URL is something: http://localhost:${port}/?token=XXXXXXXX (see your token below)

you should be able to connect to jupyter notebook running remotly on prince compute node with above url

EOF

unset XDG_RUNTIME_DIR
if [ "$SLURM_JOBTMP" != "" ]; then
    export XDG_RUNTIME_DIR=$SLURM_JOBTMP
fi

jupyter notebook --no-browser --port $port --notebook-dir=$(pwd)