CWL workflow with breakpoints for debugging

hello-world-with-debug.cwl

#!/usr/bin/env cwltool
cwlVersion: v1.2
class: CommandLineTool
inputs:
  message:
    type: string
outputs:
  outfile:
    type: File
    outputBinding:
      glob: output.txt
baseCommand: [bash, run.sh]
requirements:
  DockerRequirement:
    dockerImageId: hello-world
    dockerFile: |
      FROM python
  InitialWorkDirRequirement: 
    listing:
      - entryname: inputs.json
        entry: $(inputs)
      - entryname: cwl.py
        entry: |
          #!/usr/bin/env python
          import os
          import sys
          import time

          if sys.argv[1] == "pause":
            # normal print will be catched by cwltool
            with open("state", "a") as f:
              f.write("PAUSED")

            os.system(f"echo Debug mode enabled and workflow paused.")
            os.system(f"echo Access container: docker exec -it {os.environ["HOSTNAME"]} bash")
            os.system(f"echo Unpause container: docker exec -it {os.environ["HOSTNAME"]} python cwl.py next")
            
            paused = True
            while paused:
              time.sleep(1)
              paused = open("state", "r").read() == "PAUSED"
          elif sys.argv[1] == "next":
            with open("state", "a") as f:
              f.write("RUN")
      - entryname: run.sh
        entry: |
          #!/usr/bin/env bash
          echo hi
          python cwl.py pause
          echo hello, $(inputs.message) > output.txt

Run with cwltool hello-world-with-debug.cwl --message world . Remove python cwl.py pause to remove the break point.

Nice trick! I just mentioned it on the CWL forum: https://cwl.discourse.group/t/tool-activate/1042/3?u=mrc

I could imagine a version of this that replaced the Python script with a POSIX shell script, making it usable with even more containers.

And as mentioned by the original poster at the link above, this technique could be adapted into a feature implemented by cwltool and other CWL implementations to generically "enter" or "activate" the environment of a CWL CommandLineTool for learning, troubleshooting or debugging; without having to edit the CWL descriptionsl files themselves.

Good catch, thanks! It's both a stupid hack and a clean solution at the same time.

Essentially:

• Generate a new document based on the given CommandLineTool: adding a script to the InitialWorkDirRequirement entries and replacing the basecommand with the invocation of said script. (For my use case: remove the Input and Ouput entries too)
• The tool is run though the cwltool as usual.
• The script being run by the baseCommand has two purposes:
  • communicate the execution environment's details (viz. running containerID), and
  • simply hang, it's purpose is to keep its execution environment running.
• From outside the cwltool process you connect to the execution environment (viz docker run containerID) and do your thing
• Stop the hung script.

The coordination mechanism between the hanging script and the 'outside' is by local file (inside the container), as stdout/stderr is redirected by the cwltool. I assume what would also work in my case is to SIGSTOP/SIGCONT the script process, unless that gets caught by cwltool's container runner.

Yes, everything is a file ;) If we port the Python script to POSIX, it'd be the most general way to have "breakpoints" in almost any running container without modifying the initial docker image.