jobTree_failedLoop

Logging set at level: DEBUG
Logging to file: None
The job tree appears to already exist, so we'll reload it
Written the config file
Got the maxCPU: 0
Setting up the thread pool with 60 threads given the max threads 60 and the max cpus 9000
Reloaded the jobtree
Written the environment for the jobs to the environment file
Got parameters,rescue jobs frequency: 1800.0 max job duration: 9.22337203685e+18
Checked batch system has no running jobs and no updated jobs
Found 1 jobs to start and 1 parent jobs with children to run
Starting the main loop
Built the jobs list, currently have 1 jobs to update and 0 jobs issued
Job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job has a new command that we can now issue
Issued the job command: /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: 0 
Issued the job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: (1, 0) and cpus: 1
sbatch command sbatch /n/regal/edwards_lab/ratites/wga/ratite_final/slurm_scripts/tmpNFN76s.sbatch
-- mem : 2147
-- ntasks : 1
-- partition : serial_requeue
-- command : /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
-- time : 540

Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is RUNNING
Status of jobid 38197532 is CANCELLED
Batch system is reporting that the job (1, 0) /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job failed with exit value 1
No log file is present, despite job failing: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
Due to failure we are reducing the remaining retry count of job /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to 5
We have set the default memory of the failed job to 2147483648 bytes
Added job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to active jobs
Built the jobs list, currently have 1 jobs to update and 0 jobs issued
Job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job has a new command that we can now issue
Issued the job command: /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: 1 
Issued the job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: (1, 1) and cpus: 1
sbatch command sbatch /n/regal/edwards_lab/ratites/wga/ratite_final/slurm_scripts/tmp8e7oWF.sbatch
-- mem : 2147
-- ntasks : 1
-- partition : serial_requeue
-- command : /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
-- time : 540

Status of jobid 38198146 is PENDING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is RUNNING
Status of jobid 38198146 is COMPLETING
Status of jobid 38198146 is CANCELLED
Batch system is reporting that the job (1, 1) /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job failed with exit value 1
No log file is present, despite job failing: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
Due to failure we are reducing the remaining retry count of job /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to 5
We have set the default memory of the failed job to 2147483648 bytes
Added job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to active jobs
Built the jobs list, currently have 1 jobs to update and 0 jobs issued
Job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job has a new command that we can now issue
Issued the job command: /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: 2 
Issued the job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: (1, 2) and cpus: 1
sbatch command sbatch /n/regal/edwards_lab/ratites/wga/ratite_final/slurm_scripts/tmpRKAj9T.sbatch
-- mem : 2147
-- ntasks : 1
-- partition : serial_requeue
-- command : /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
-- time : 540

Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is RUNNING
Status of jobid 38198627 is COMPLETING
Status of jobid 38198627 is CANCELLED
Batch system is reporting that the job (1, 2) /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job failed with exit value 1
No log file is present, despite job failing: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
Due to failure we are reducing the remaining retry count of job /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to 5
We have set the default memory of the failed job to 2147483648 bytes
Added job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job to active jobs
Built the jobs list, currently have 1 jobs to update and 0 jobs issued
Job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job has a new command that we can now issue
Issued the job command: /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: 3 
Issued the job: /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job with job id: (1, 3) and cpus: 1
sbatch command sbatch /n/regal/edwards_lab/ratites/wga/ratite_final/slurm_scripts/tmpmhy8MJ.sbatch
-- mem : 2147
-- ntasks : 1
-- partition : serial_requeue
-- command : /n/home12/tsackton/cuff/progressiveCactus/python/bin/python -E /n/home12/tsackton/cuff/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py /n/home12/tsackton/cuff/progressiveCactus/submodules /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree /n/home12/tsackton/ratite_scratch/wga/ratite_final/ratiteDir/jobTree/jobs/t1/job
-- time : 540

Status of jobid 38199072 is RUNNING
Status of jobid 38199072 is RUNNING
Status of jobid 38199072 is RUNNING
Status of jobid 38199072 is RUNNING
Status of jobid 38199072 is RUNNING