Useful slurm commands and tricks

useful_slurm.md

Cancel all my jobs

scancel -u <USERNAME>

Set time limit for one job through env variables

Precedence: command line options > environment variables > options set in a batch script

Reference

export SBATCH_TIMELIMIT=05:00:00
sbatch <bash filename>

Attach to the compute node of one running job

Useful to check out the status of the machine

srun --pty --overlap --jobid $JOBID /bin/bash

Pass all commandline arguments into train.py

In run.bash, write

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.out
#SBATCH --mem=32G
#SBATCH --cpus-per-task=8

$*

Then we can use following to run python using any parameters we want. For example

sbatch ./run.bash python --dataset cifar10 --model resnet