Create and submit a slurm job to create a conda environment for jupyter notebook
rico@desmond:~$ MY_TEST_ENVIRONMENT=wtest
rico@desmond:~$ MY_TEST_DIR=/galaxy/home/$USER/$MY_TEST_ENVIRONMENT
rico@desmond:~$ test -e $MY_TEST_DIR || mkdir -p $MY_TEST_DIR
rico@desmond:~$ cat << EOF > $MY_TEST_DIR/make-conda-environment.job
> #!/usr/bin/env bash
> #SBATCH --export=HOME=/home/$USER
> #SBATCH --workdir=$MY_TEST_DIR
> #SBATCH --output=make-conda-environment.out
> #SBATCH --ntasks=1
> #SBATCH --cpus-per-task=1
> #SBATCH --nodes=1
> #SBATCH --mem=32G
>
> MY_CONDA_DIR="\$HOME/conda"
> test -e "\$MY_CONDA_DIR" || mkdir -p "\$MY_CONDA_DIR"
>
> export CONDA_ENVS_PATH="\$MY_CONDA_DIR"
>
> conda create --yes --name wtest 'python>=3.5' 'r-base>=3.4' 'rpy2>=2.9' jupyter pandas simplegeneric 'icu>=58'
>
> exit 0
> EOF
rico@desmond:~$ sbatch $MY_TEST_DIR/make-conda-environment.job
Submitted batch job 310838
Create and submit slurm job to start jupyter notebook
rico@desmond:~$ MY_JUPYTER_PORT=8900
rico@desmond:~$ cat << EOF > $MY_TEST_DIR/start-jupyter-notebook.job
> #!/usr/bin/env bash
> #SBATCH --export=HOME=/home/$USER
> #SBATCH --workdir=$MY_TEST_DIR
> #SBATCH --output=start-jupyter-notebook.out
> #SBATCH --ntasks=1
> #SBATCH --cpus-per-task=1
> #SBATCH --nodes=1
> #SBATCH --mem=32G
>
> MY_CONDA_DIR="\$HOME/conda"
>
> export CONDA_ENVS_PATH="\$MY_CONDA_DIR"
> source activate $MY_TEST_ENVIRONMENT
> jupyter notebook --no-browser --ip=0.0.0.0 --port=$MY_JUPYTER_PORT
>
> source deactivate
>
> exit 0
> EOF
rico@desmond:~$ sbatch $MY_TEST_DIR/start-jupyter-notebook.job
Submitted batch job 310840
After the slurm job starts, look in the output file for connection information
rico@desmond:~$ cat $MY_TEST_DIR/start-jupyter-notebook.out
[I 13:22:17.366 NotebookApp] Serving notebooks from local directory: /nfs/thumper.galaxyproject.org/home/rico/wtest
[I 13:22:17.367 NotebookApp] The Jupyter Notebook is running at:
[I 13:22:17.367 NotebookApp] http://(c14 or 127.0.0.1):8900/?token=09079e39fc1ced5b2a837ce5cab6f6599e1750711bcb0035
[I 13:22:17.367 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:22:17.380 NotebookApp]
To access the notebook, open this file in a browser:
file:///nfs/thumper.galaxyproject.org/home/rico/.local/share/jupyter/runtime/nbserver-5781-open.html
Or copy and paste one of these URLs:
http://(c14 or 127.0.0.1):8900/?token=09079e39fc1ced5b2a837ce5cab6f6599e1750711bcb0035
At this point, you can log out of desmond. The jupyter notebook server is now running on one of the compute nodes (c14 in this example).
From our local machine (desktop, laptop, whatever), we can create an ssh tunnel to connect to the jupyter notebook server running on the slurm cluster.
nexus6:~ rico$ MY_JUPYTER_PORT=8900
nexus6:~ rico$ ssh -N -f -L localhost:$MY_JUPYTER_PORT:c14.c.bx.psu.edu:$MY_JUPYTER_PORT rico@uniport.bx.psu.edu
Password:
Now we can open a browser on our local machine and direct the brower to http://localhost:$MY_JUPYTER_PORT (http://localhost:8900/?token=09079e39fc1ced5b2a837ce5cab6f6599e1750711bcb0035 in this example)
An example of using rpy2 can be seen next
To shut down your jupyter notebook server and end your slurm job, click on the "Quit" button
