Skip to content

Instantly share code, notes, and snippets.

@willirath
Last active December 11, 2023 14:15
Show Gist options
  • Save willirath/2176a9fa792577b269cb393995f43dda to your computer and use it in GitHub Desktop.
Save willirath/2176a9fa792577b269cb393995f43dda to your computer and use it in GitHub Desktop.
Dask-Jobqueue SLURMCluster with Singularity
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@willirath
Copy link
Author

  1. Can you SSH back into the host machine? The error could stem from ssh $(hostname) not being possible (at least without a password). You'd need to set up SSH keys for this.

@thomasarsouze
Copy link

Thx for your quick answer. Indeed, thanks to a local security feature provided by our admins, ssh $(hostname) is refused ! Will resume once they remove that.

@kathoef
Copy link

kathoef commented Mar 9, 2021

We have come up with a little convenience tool that provides a structured way of bind mounting host-system SLURM libraries into a Singularity container session and thus enables the batch scheduler commands. This approach omits the SSH restrictions that system administrators might have set up (we also use such an HPC system, which has motivated that development).

All you need is to come up with a system-specific "configuration file" (which needs basically a one-time exploratory session with a few strace commands to isolate the necessary batch scheduler shared libraries and configuration files). Make sure you have read the compatibility section, though, as there are a few limitations: https://github.com/ExaESM-WP4/Batch-scheduler-Singularity-bindings

/cc @vsoch and @thomasarsouze

@poplarShift
Copy link

This is great, thank you so much for this! Only needed minor tweaks to account for differences in my cluster and the singularity version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment