Run multiple jobs with a max number of parallel processes

parallel_jobs.md

This is a function to run multiple jobs in parallel, but with a max number of parallel processes. New jobs will be run only if some previous jobs end and free up resources.

Credit to GPT-4.

First declare the following function in bash

function wait_for_jobs() {
  local num_jobs=$1
  while true; do
    current_jobs=$(jobs -p | wc -l)
    if [ "$current_jobs" -lt "$num_jobs" ]; then
      break
    fi
    sleep 1
  done
}

Then run command as follows

# Maximum number of parallel processes
max_jobs=12

# Change the arguments and actual program to your own
for ARG in 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1
do
  wait_for_jobs $max_jobs # This line will block if there are already $max_jobs processes
  echo $ARG
  python train.py --arg $ARG & # Don't forget the "&" sign. 
done