links for Singularity and Cromwell

ohlordwhy.md

Quick Links

These are some quick links for the singularity work for Cromwell! Because slack + markdown == pain

• here is the container recipe
• docs for building and running
• the backend configuration with a struggle bus for me to get a PID for the running container :P
• singularity workflow opts (must build container locally since not on Shub!
• possible avenue to get singularity in singularity

Script submit:

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script.submit 
#!/bin/bash
chmod u+x /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script && \
singularity \
exec \
gemBS.simg \
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/0df0d373-03b3-42f3-9a75-c51ad60392bc/call-flatten_/execution/script

return code

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/rc

out

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/stdout
flowcell_1_1_1.fastq.gz
flowcell_1_1_2.fastq.gz

I think it's running/working because the files in mapping... are there

$ ls /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/mapping/
flowcell_1_1_1.fastq.gz  flowcell_1_1_2.fastq.gz

write lines

$ cat /home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/execution/write_lines_841eb9521a4aaf490c4ce273954daeae.tmp 
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/inputs/-1629611677/flowcell_1_1_1.fastq.gz
/home/vanessa/Documents/Dropbox/Code/labs/cherry/pipelines/wgbs-pipeline/cromwell-executions/wgbs/2ea00c38-4ee4-4155-9f85-14f55beb8982/call-flatten_/inputs/-1629611677/flowcell_1_1_2.fastq.gz

All the stderr files are... empty! :P

I just added one more runtime configuration - that will start and execute a command to an instance instead. This might be an easier avenue to get a process id (e.g., singularity instance list but it still seems to run ok, and exit without any obvious error, so please take a look! The command for the instance run would be:

$ java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity-instance cromwell-34.jar run runners/test.wdl -i data/TEST-YEAST/inputs.json -o workflow_opts/singularity.json

And it uses the same container, but as an instance (singularity-instance is the backend.conf section)