How to user torque openpbs in python, a small howto

1 how_to_use_torque_openpbs_in_python_subprocess

Just a small note, here is how you can convert commandline calls in python to use openpbs/torque
If the command is doing something in the background without any dependency to the main script thats easy,
just use qsub in your subprocess call and let if fork off.

But what you do if you have some subprocess calls which wait for a program or script to finish (it depends on it)?
After some Internet research I found qsub's parameter "-I -x"  to be exactly what I needed!

lets say you have the following command fragment:

cmd = 'lastdb -Q 0 %s %s' % (last_index, validated_ref)  # format the database
proc = subprocess.Popen(cmd, shell=True)
status = proc.wait()


Here is the solution how this can be written to use openpbs using its interactive mode (-I) but you need to take care that no interactive shell 
will be openend (-x)

Here is how to replace the code:

scriptDir = os.path.dirname(os.path.realpath(__file__))
cmd = 'lastdb -Q 0 %s %s' % (last_index, validated_ref)  # format the database
cmd = "qsub -I -x -v CMD='" + cmd + "' -N 'last_index_build' " + scriptDir + "/openpbs.sh"
proc = subprocess.Popen(cmd, shell=True)
status = proc.wait()

2 openpbs.sh

#PBS -S /bin/bash
#PBS -V 
# put in here other stuff e.g. #PBS -l nodes=1:ppn=5
# put in here other stuff e.g. #PBS -l mem=4gb


$CMD 2> /dev/null

3 advanced_example_torque_pbs_in_python

Here is an example I came across which includes some more advanced replacement technique:

If you have python code which runs a subprocess and pipes some data into it e.g.

read = '>%s \r\n%s' % (seqid, fastqhash['seq'])
cmd = 'bwa mem -x ont2d %s %s -' % (options,
            ref_fasta_hash[dbname]['bwa_index'])
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE,
                            stdin=subprocess.PIPE, shell=True)
(out, err) = proc.communicate(input=read)
status = proc.wait()

You need to work with files instead of pipes because I have not found an easy mechanism to pipe data into a qsub command script on the shell. 
Discussions can be found here -http://stackoverflow.com/questions/18322286/how-can-i-use-a-pipe-or-redirect-in-a-qsub-command
But I think it is more easy to write the content into a temp file e.g.

bwaTmpDir = os.path.dirname(os.path.realpath(ref_fasta_hash[dbname]['bwa_index']))
f = tempfile.NamedTemporaryFile(dir=bwaTmpDir, delete=False)
f.write(read)
f.close()
scriptDir = os.path.dirname(os.path.realpath(__file__))
cmd = 'bwa mem -x ont2d %s %s %s' % (options,
            ref_fasta_hash[dbname]['bwa_index'], f.name)
cmd = ["qsub", "-I", "-x", "-v", "CMD=" + cmd, "-N", "'run_bwa_align'",  scriptDir + "/openpbs.sh"]
out = None
try:
    out = subprocess.check_output(cmd)
except subprocess.CalledProcessError, e:
    err = e.output
    
# There is a problem with this approach though, have you found out?
# there is no parameter to tell qsub to supress all internal output when using -I -x so the output
# will always contain qsub related stuff and unnecessary newlines, e.g.

qsub: job 777.xxx.xxx.de ready

qsub: job 777.xxx.xxx.de completed


Quick fix is to remove those qsub: lines and empty lines or write the output of the qsub command to a file, check out my class 
pbs_utils.py which does all of that.