HMDC / IQSS RCE Notes to run python batch jobs with condor

00_notes.md

Submitting a Batch Job

Use python.submit to submit a script to condor with condor_submit python.submit.

Edit the file names to match what you'd normally use.

Change the input script to list_installed_packages.py to view all the installed packages / versions. View the out file.

HMDC cluster has these as of Sep 26 2014:

[‘argparse==1.2.1’, ‘babel==0.9.6’, ‘backports.ssl-match-hostname==3.4.0.2’,
‘beautifulsoup4==4.3.2’, ‘certifi==14.05.14’, ‘docutils==0.11’,
‘elementtree==1.2.6-20050316’, ‘feedparser==5.1.3’, ‘humanize==0.5’,
‘icalendar==3.8.3’, ‘importlib==1.0.3’, ‘ipython==2.2.0’, ‘jinja2==2.6’,
‘lxml==3.4.0’, ‘markupsafe==0.11’, ‘matplotlib==1.3.1’, ‘mechanize==0.2.5’,
‘mongo==0.2.0’, ‘nltk==3.0.0’, ‘nose==1.3.0’, ‘nosexml==0.19’,
‘nosexunitlite==0.3.3’, ‘numpy==1.8.2’, ‘pandas==0.14.1’, ‘pep8==1.5.7’,
‘pycurl2==7.20.0’, ‘pygments==1.5’, ‘pymongo==2.7.2’, ‘pyparsing==2.0.2’,
‘python-dateutil==2.2’, ‘pytz==2014.7’, ‘rpy2==2.4.3’, ‘scikit-learn==0.15.2’,
‘scipy==0.14.0’, ‘selenium==2.43.0’, ‘simplejson==3.2.0’, ‘six==1.8.0’,
‘sphinx==1.1.3’, ‘sqlalchemy==0.7.9’, ‘termcolor==1.1.0’, ‘tornado==4.0.2’,
‘virtualenv==1.10.1’, ‘werkzeug==0.8.3’, ‘wsgiref==0.1.2’]

Running IPython Notebook interactively on a compute node

Interactive sessions are different than batch jobs in that you can have a process running and, well, interact with it. The interactive tasks also seem to have a higher RAM cap, versus batch, I think, is limited to 4GB.

For some reason I couldn't get the installed ipython to work but installing with anaconda works great. Also, you can't seem to access the network within the node but you can just run the browser within and it works okay:

• In the Applications menu > RCE Powered Applications run RCE Shell
• run screen -R (or byobu or tmux, more about GNU screen: http://aperiodic.net/screen/start)
• run wget http://repo.continuum.io/miniconda/Miniconda-3.7.0-Linux-x86_64.sh (Installer URL from: http://conda.pydata.org/miniconda.html)
• run chmod +x Miniconda-3.7.0....
• run ./Miniconda-3.7.0....
• run conda install ipython tornado jinja2 pyzmq plus whatever other packages you need, like numpy, pandas, pytables etc.
• run ipython notebook --no-browser.
• Create a new window within your terminal muxer (Ctrl-A C for screen) and run firefox, and navigate to the URL that ipython notebook gave. (To switch between windows, try Ctrl-A N).

More notes

• http://projects.iq.harvard.edu/files/rce/files/rce_workshop_-_batch_processing.pdf
• RCE docs: http://projects.iq.harvard.edu/rce/book/rce-docs

list_installed_packages.py

import pip
installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version) for i in installed_packages])
print installed_packages_list

python.submit

Universe        = vanilla
Executable      = /usr/local/bin/python27
should_transfer_files = NO
Requirements = Memory >= 256
input   = /nfs/home/M/makmanalp/test.py
output  = /nfs/home/M/makmanalp/condor_submit_util/out
error   = /nfs/home/M/makmanalp/condor_submit_util/error
Log     = /nfs/home/M/makmanalp/condor_submit_util/log
Queue   1