links for Singularity and Cromwell

ohlordwhy.md

Quick Links

These are some quick links for the singularity work for Cromwell! Because slack + markdown == pain

• here is the container recipe
• docs for building and running
• the backend configuration with a struggle bus for me to get a PID for the running container :P
• singularity workflow opts (must build container locally since not on Shub!
• possible avenue to get singularity in singularity

Overall I (think) the structure is there to work, and the issue (at least one that pops up in the interface) is getting the PID for the container. It might be something else too, I'm hoping someone with experience can take a look!

Script submit:

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script.submit 
#!/bin/bash
chmod u+x /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script && \
singularity \
exec \
gemBS.simg \
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script

return code

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/rc

out

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/stdout
flowcell_1_1_1.fastq.gz
flowcell_1_1_2.fastq.gz

I think it's running/working because the files in mapping... are there

$ ls /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/mapping/
flowcell_1_1_1.fastq.gz  flowcell_1_1_2.fastq.gz

write lines

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/write_lines_841eb9521a4aaf490c4ce273954daeae.tmp 
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/inputs/-1629611677/flowcell_1_1_1.fastq.gz
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/inputs/-1629611677/flowcell_1_1_2.fastq.gz

All the stderr files are... empty! :P

I just added one more runtime configuration - that will start and execute a command to an instance instead. This might be an easier avenue to get a process id (e.g., singularity instance list but it still seems to run ok, and exit without any obvious error, so please take a look! The command for the instance run would be:

$ java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity-instance cromwell-34.jar run runners/test.wdl -i data/TEST-YEAST/inputs.json -o workflow_opts/singularity.json

And it uses the same container, but as an instance (singularity-instance is the backend.conf section)