Skip to content

Instantly share code, notes, and snippets.

View georgegu1997's full-sized avatar

Qiao Gu georgegu1997

View GitHub Profile
@georgegu1997
georgegu1997 / parallel_jobs.md
Created June 14, 2023 03:46
Run multiple jobs with a max number of parallel processes

This is a function to run multiple jobs in parallel, but with a max number of parallel processes. New jobs will be run only if some previous jobs end and free up resources.

Credit to GPT-4.

First declare the following function in bash

function wait_for_jobs() {
  local num_jobs=$1
 while true; do
@georgegu1997
georgegu1997 / useful_slurm.md
Last active February 2, 2024 01:17
Useful slurm commands and tricks

Cancel all my jobs

scancel -u <USERNAME>

Set time limit for one job through env variables

Precedence: command line options > environment variables > options set in a batch script

Reference