Skip to content

Instantly share code, notes, and snippets.

@astrofrog
Created May 2, 2013 11:39
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
  • Save astrofrog/5501664 to your computer and use it in GitHub Desktop.
Save astrofrog/5501664 to your computer and use it in GitHub Desktop.
Submitting jobs via qsub in Python
import os
import random
import string
import tempfile
import subprocess
def random_id(length=8):
return ''.join(random.sample(string.ascii_letters + string.digits, length))
TEMPLATE_SERIAL = """
#####################################
#$ -S /bin/bash
#$ -cwd
#$ -N {name}
#$ -e {errfile}
#$ -o {logfile}
#$ -pe make {slots}
#####################################
echo "------------------------------------------------------------------------"
echo "Job started on" `date`
echo "------------------------------------------------------------------------"
{script}
echo "------------------------------------------------------------------------"
echo "Job ended on" `date`
echo "------------------------------------------------------------------------"
"""
def submit_python_code(code, name="job", logfile="output.$JOB_ID", errfile="error.$JOB_ID", cleanup=True, prefix="", slots=1):
base = prefix + "submit_{0}".format(random_id())
open(base + '.py', 'wb').write(code)
script = "python " + base + ".py"
open(base + '.qsub', 'wb').write(TEMPLATE_SERIAL.format(script=script, name=name, logfile=logfile, errfile=errfile, slots=slots))
try:
subprocess.call('qsub < ' + base + '.qsub', shell=True)
finally:
if cleanup:
os.remove(base + '.qsub')
@astrofrog
Copy link
Author

In [2]: qsub.submit_python_code("hyperion model.rtin model.rtout", name='model', logfile='model.log', errfile='model.err')
Your job 59683 ("model") has been submitted

@astrofrog
Copy link
Author

Remove this line for serial runs:

#$ -pe make {slots}

@sassan72
Copy link

I have a script file bh1.py. I am going to run it with qsub. how can I run my file with your qsub.py code?

@sassan72
Copy link

I have a bh1.py script. i am going to use your code to submit my job. how can I do it. I have basic knowledge about PSB and cluster performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment