Skip to content

Instantly share code, notes, and snippets.

@gmgitx
Created September 19, 2018 02:56
Show Gist options
  • Save gmgitx/db14eea62974973e336de4ff1f90c43c to your computer and use it in GitHub Desktop.
Save gmgitx/db14eea62974973e336de4ff1f90c43c to your computer and use it in GitHub Desktop.
```
[2018-09-19 02:34:46,62] [info] Running with database db.url = jdbc:hsqldb:mem:5c60b7f9-5aef-406c-9ad2-9ff13776de58;shutdown=false;hsqldb.tx=mvcc
[2018-09-19 02:35:04,66] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2018-09-19 02:35:04,69] [info] [RenameWorkflowOptionsInMetadata] 100%
[2018-09-19 02:35:05,18] [info] Running with database db.url = jdbc:hsqldb:mem:2dbdec8e-bc18-4085-b951-aaf4371a1be5;shutdown=false;hsqldb.tx=mvcc
[2018-09-19 02:35:05,81] [warn] This actor factory is deprecated. Please use cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory for PAPI v1 or cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory for PAPI v2
[2018-09-19 02:35:05,86] [warn] Couldn't find a suitable DSN, defaulting to a Noop one.
[2018-09-19 02:35:05,87] [info] Using noop to send events.
[2018-09-19 02:35:06,19] [info] Slf4jLogger started
[2018-09-19 02:35:06,47] [info] Workflow heartbeat configuration:
{
"cromwellId" : "cromid-c4f43a3",
"heartbeatInterval" : "2 minutes",
"ttl" : "10 minutes",
"writeBatchSize" : 10000,
"writeThreshold" : 10000
}
[2018-09-19 02:35:06,72] [info] Metadata summary refreshing every 2 seconds.
[2018-09-19 02:35:06,98] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-19 02:35:06,99] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-09-19 02:35:07,02] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-19 02:35:10,60] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-09-19 02:35:10,68] [info] SingleWorkflowRunnerActor: Version 34
[2018-09-19 02:35:10,69] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-09-19 02:35:10,78] [info] PAPIQueryManager Running with 3 workers
[2018-09-19 02:35:10,78] [info] JES batch polling interval is 33333 milliseconds
[2018-09-19 02:35:10,79] [info] JES batch polling interval is 33333 milliseconds
[2018-09-19 02:35:10,87] [info] JES batch polling interval is 33333 milliseconds
[2018-09-19 02:35:10,87] [info] Unspecified type (Unspecified version) workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 submitted
[2018-09-19 02:35:11,00] [info] SingleWorkflowRunnerActor: Workflow submitted 2a0ed041-872c-4e5f-b2b5-54735758a707
[2018-09-19 02:35:11,01] [info] 1 new workflows fetched
[2018-09-19 02:35:11,01] [info] WorkflowManagerActor Starting workflow 2a0ed041-872c-4e5f-b2b5-54735758a707
[2018-09-19 02:35:11,02] [info] WorkflowManagerActor Successfully started WorkflowActor-2a0ed041-872c-4e5f-b2b5-54735758a707
[2018-09-19 02:35:11,02] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-09-19 02:35:11,04] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-09-19 02:35:11,05] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-09-19 02:35:11,19] [info] MaterializeWorkflowDescriptorActor [2a0ed041]: Parsing workflow as WDL draft-2
[2018-09-19 02:35:31,05] [info] MaterializeWorkflowDescriptorActor [2a0ed041]: Call-to-Backend assignments: atac.bowtie2 -> slurm, atac.ataqc -> slurm, atac.macs2_pooled -> slurm, atac.idr_pr -> slurm, atac.overlap -> slurm, atac.bam2ta -> slurm, atac.macs2_pr2 -> slurm, atac.xcor -> slurm, atac.filter -> slurm, atac.macs2 -> slurm, atac.pool_ta -> slurm, atac.reproducibility_idr -> slurm, atac.spr -> slurm, atac.read_genome_tsv -> slurm, atac.macs2_ppr2 -> slurm, atac.macs2_pr1 -> slurm, atac.qc_report -> slurm, atac.idr_ppr -> slurm, atac.pool_ta_pr2 -> slurm, atac.overlap_ppr -> slurm, atac.trim_adapter -> slurm, atac.idr -> slurm, atac.pool_ta_pr1 -> slurm, atac.overlap_pr -> slurm, atac.reproducibility_overlap -> slurm, atac.macs2_ppr1 -> slurm
[2018-09-19 02:35:31,40] [warn] slurm [2a0ed041]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,43] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,44] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,45] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,48] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,49] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,49] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,50] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,51] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:31,52] [warn] slurm [2a0ed041]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.read_genome_tsv
[2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: 'enable_idr'. Running conditional section
[2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!align_only && !true_rep_only && enable_idr'. Running conditional section
[2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!align_only && !true_rep_only'. Running conditional section
[2018-09-19 02:35:33,80] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!disable_xcor'. Running conditional section
[2018-09-19 02:35:33,83] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: 'enable_idr'. Running conditional section
[2018-09-19 02:35:33,84] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Condition met: '!true_rep_only'. Running conditional section
[2018-09-19 02:35:34,71] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks
[2018-09-19 02:35:36,66] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: cat /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/inputs/378634365/hg19.tsv
[2018-09-19 02:35:36,72] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: executing: sbatch \
--export=ALL \
-J cromwell_2a0ed041_read_genome_tsv \
-D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv \
-o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/stdout \
-e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/stderr \
-t 60 \
-n 1 \
--ntasks-per-node=1 \
--cpus-per-task=1 \
--mem=4000 \
-p broadwl \
\
--wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-read_genome_tsv/execution/script"
[2018-09-19 02:35:37,08] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: job id: 49805408
[2018-09-19 02:35:37,09] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCodeFile
[2018-09-19 02:35:38,60] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.trim_adapter (2 shards)
[2018-09-19 02:35:38,63] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Unrecognized runtime attribute keys: disks
[2018-09-19 02:35:38,64] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Unrecognized runtime attribute keys: disks
[2018-09-19 02:35:38,94] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: python $(which encode_trim_adapter.py) \
/project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/write_tsv_da8046fb49e6a4af98b0ba5e9b4e96ab.tmp \
--adapters /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
--paired-end \
--auto-detect-adapter \
--min-trim-len 5 \
--err-rate 0.1 \
--nth 1
[2018-09-19 02:35:38,95] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: python $(which encode_trim_adapter.py) \
/project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/write_tsv_0052698ca1a6bc6408797374c5d078e3.tmp \
--adapters /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
--paired-end \
--auto-detect-adapter \
--min-trim-len 5 \
--err-rate 0.1 \
--nth 1
[2018-09-19 02:35:38,98] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: executing: sbatch \
--export=ALL \
-J cromwell_2a0ed041_trim_adapter \
-D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0 \
-o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/stdout \
-e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/stderr \
-t 1440 \
-n 1 \
--ntasks-per-node=1 \
--cpus-per-task=1 \
--mem=12000 \
-p broadwl \
\
--wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-0/execution/script"
[2018-09-19 02:35:38,98] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: executing: sbatch \
--export=ALL \
-J cromwell_2a0ed041_trim_adapter \
-D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1 \
-o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/stdout \
-e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/stderr \
-t 1440 \
-n 1 \
--ntasks-per-node=1 \
--cpus-per-task=1 \
--mem=12000 \
-p broadwl \
\
--wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-trim_adapter/shard-1/execution/script"
[2018-09-19 02:35:42,02] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: job id: 49805411
[2018-09-19 02:35:42,03] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: job id: 49805410
[2018-09-19 02:35:42,12] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile
[2018-09-19 02:35:42,13] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Status change from - to WaitingForReturnCodeFile
[2018-09-19 02:36:15,21] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done
[2018-09-19 02:37:40,90] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done
[2018-09-19 02:37:42,15] [info] WorkflowExecutionActor-2a0ed041-872c-4e5f-b2b5-54735758a707 [2a0ed041]: Starting atac.bowtie2
[2018-09-19 02:37:42,62] [warn] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: Unrecognized runtime attribute keys: preemptible, disks
[2018-09-19 02:37:42,67] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: python $(which encode_bowtie2.py) \
/project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/646531885/male.hg19.fa.tar \
/project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/362506696/merge_fastqs_R1_ENCFF341MYG.subsampled.400.trim.merged.fastq.gz /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/inputs/362506696/merge_fastqs_R2_ENCFF248EJF.subsampled.400.trim.merged.fastq.gz \
--paired-end \
--multimapping 4 \
\
--nth 2
[2018-09-19 02:37:42,72] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.bowtie2:0:1]: executing: sbatch \
--export=ALL \
-J cromwell_2a0ed041_bowtie2 \
-D /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0 \
-o /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stdout \
-e /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stderr \
-t 2880 \
-n 1 \
--ntasks-per-node=1 \
--cpus-per-task=2 \
--mem=16000 \
-p broadwl \
\
--wrap "/bin/bash /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/script"
[2018-09-19 02:38:01,41] [info] DispatchedConfigAsyncJobExecutionActor [2a0ed041atac.trim_adapter:1:1]: Status change from WaitingForReturnCodeFile to Done
[2018-09-19 02:38:02,71] [error] WorkflowManagerActor Workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 failed (during ExecutingWorkflowState): java.lang.RuntimeException: Unable to start job. Check the stderr file for possible errors: /project2/name1/name2/ASP/atac-seq-pipeline/cromwell-executions/atac/2a0ed041-872c-4e5f-b2b5-54735758a707/call-bowtie2/shard-0/execution/stderr.submit
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.$anonfun$execute$2(SharedFileSystemAsyncJobExecutionActor.scala:131)
at scala.util.Either.fold(Either.scala:188)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:126)
at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:208)
at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:600)
at scala.util.Try$.apply(Try.scala:209)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:600)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:600)
at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:208)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:915)
at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:907)
at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:208)
at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
at cromwell.backend.async.AsyncBackendJobExecutionActor.withRetry(AsyncBackendJobExecutionActor.scala:61)
at cromwell.backend.async.AsyncBackendJobExecutionActor.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:65)
at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:88)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at cromwell.backend.impl.sfs.config.DispatchedConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:208)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
at akka.actor.ActorCell.invoke(ActorCell.scala:557)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2018-09-19 02:38:02,83] [info] WorkflowManagerActor WorkflowActor-2a0ed041-872c-4e5f-b2b5-54735758a707 is in a terminal state: WorkflowFailedState
[2018-09-19 02:38:27,85] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-09-19 02:38:32,09] [info] Workflow polling stopped
[2018-09-19 02:38:32,20] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-09-19 02:38:32,20] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-09-19 02:38:32,25] [info] Aborting all running workflows.
[2018-09-19 02:38:32,46] [info] WorkflowStoreActor stopped
[2018-09-19 02:38:32,51] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-09-19 02:38:32,51] [info] WorkflowLogCopyRouter stopped
[2018-09-19 02:38:32,51] [info] JobExecutionTokenDispenser stopped
[2018-09-19 02:38:32,51] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-09-19 02:38:32,51] [info] Connection pools shut down
[2018-09-19 02:38:32,52] [info] WorkflowManagerActor stopped
[2018-09-19 02:38:32,52] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-09-19 02:38:32,52] [info] SubWorkflowStoreActor stopped
[2018-09-19 02:38:32,52] [info] WorkflowManagerActor All workflows finished
[2018-09-19 02:38:32,55] [info] DockerHashActor stopped
[2018-09-19 02:38:32,55] [info] IoProxy stopped
[2018-09-19 02:38:32,55] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-09-19 02:38:32,55] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-09-19 02:38:32,55] [info] JobStoreActor stopped
[2018-09-19 02:38:32,55] [info] CallCacheWriteActor stopped
[2018-09-19 02:38:32,55] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-09-19 02:38:32,56] [info] ServiceRegistryActor stopped
[2018-09-19 02:38:32,63] [info] Database closed
[2018-09-19 02:38:32,63] [info] Stream materializer shut down
Workflow 2a0ed041-872c-4e5f-b2b5-54735758a707 transitioned to state Failed
[2018-09-19 02:38:32,73] [info] Automatic shutdown of the async connection
[2018-09-19 02:38:32,73] [info] Gracefully shutdown sentry threads.
[2018-09-19 02:38:32,73] [info] Shutdown finished.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment