Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Poor man's parallel execution on servers
#!/usr/bin/env bash
set -e
for N in {1..5}
do
for M in {0.1,1,2,5,10};
do
# Replace ./my_prog.py with your program to run it in parallel
./my_prog.py -N $N -M $M > output/stdout_N${N}_M${M}.txt &
echo $!
done
done
## Run as:
## ./pool_man_parallel.sh > pids
## Then kill recalcitrant processes from `pids` file, if needed.
@harrymvr

This comment has been minimized.

Copy link

harrymvr commented Feb 18, 2017

Nice one! As an alternative, you could kill all the processes by killing the pid of the parent, e.g. pkill -P <parent_pid>.

Also, if you are not sure if you have any bugs in my_prog.py, and you want the script to die and stop spawning new processes if an exception happens, you can use set -e in the beginning.

@musically-ut

This comment has been minimized.

Copy link
Owner Author

musically-ut commented Feb 18, 2017

@harrymvr Thanks for the set -e idea, I've included it in the script. I didn't know about pkill -P trick, but won't it kill the shell itself?

@harrymvr

This comment has been minimized.

Copy link

harrymvr commented Feb 18, 2017

@musically-ut pkill -P does not kill the TERM, but I think that it killed the forked processes. Let me know, because I might need to change that in my scripts too 😄

@musically-ut

This comment has been minimized.

Copy link
Owner Author

musically-ut commented Feb 20, 2017

@harrymvr The PPID of the processes put in background seems to be 1, i.e. the init process:

I am not too keen on running pkill -P 1. <_<

@harrymvr

This comment has been minimized.

Copy link

harrymvr commented Feb 20, 2017

hmmm interesting. Yeah, I would pass on that as well 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.