| #!/usr/bin/env bash | |
| #SBATCH --job-name=high_throughput | |
| #SBATCH --ntasks=13 # number of tasks running at the same time | |
| #SBATCH --cpus-per-task=1 | |
| #SBATCH --mem-per-cpu=12G | |
| #SBATCH --time=1:00:00 # one hour | |
| #SBATCH --partition=cluster | |
| #SBATCH --output=high_throughput.out | |
| #SBATCH --error=high_throughput.err | |
| #SBATCH --mail-user=myname@example.com |
Allocate 3 nodes with 1 task of 24 cpus each:
salloc -N 3 -n 3 -c 24Within the allocation, make sure you have dask etc (e.g. by activating a virtual or conda env).
Start a scheduler (on the first node) and three workers (one per node):
$ srun -n1 -N1 -r0 dask-scheduler --scheduler-file scheduler.json &>> scheduler.log &