Skip to content

Instantly share code, notes, and snippets.

@ryu1kn
Last active August 14, 2021 14:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ryu1kn/13243719f672b7f81c9d9c88e010fcc7 to your computer and use it in GitHub Desktop.
Save ryu1kn/13243719f672b7f81c9d9c88e010fcc7 to your computer and use it in GitHub Desktop.
Parallel job execution and aggregated exit status with Bash

The idea is from https://unix.stackexchange.com/questions/344360/collect-exit-codes-of-parallel-background-processes-sub-shells#436932

$ ./main.sh 5
[Task 4] Job starting...
[Task 4] Sleeping...
[Task 1] Job starting...
[Task 1] Sleeping...
[Task 2] Job starting...
[Task 2] Sleeping...
[Task 5] Job starting...
[Task 5] Sleeping...
[Task 3] Job starting...
[Task 3] Sleeping...
[Task 5] Successfully completed
[Task 1] Successfully completed
[Task 2] Successfully completed
[Task 3] Something went wrong!
[Task 4] Successfully completed

$ echo $?
1

As the outputs from sub tasks are prefixed with task IDs, you can easily filter to check only particular task's output:

$ ./main.sh 5 | grep '^\[Task 3\] '
[Task 3] Job starting...
[Task 3] Sleeping...
[Task 3] Something went wrong!

Or group the output by tasks

$ ./main.sh 5 | sort -s -n -t' ' -k 2,2
[Task 1] Job starting...
[Task 1] Sleeping...
[Task 1] Successfully completed
[Task 2] Job starting...
...
#!/usr/bin/env bash
set -euo pipefail
readonly num_of_tasks=${1:-4}
pids=()
for i in $(seq 1 "$num_of_tasks") ; do
{ ./sub-task.sh "$i" 2>&1 | sed "s/^/[Task $i] / " ; } &
pids+=("$!")
done
overall_exit_code=0
for p in "${pids[@]}"; do
wait "$p" || overall_exit_code=$((overall_exit_code + $?))
done
exit $overall_exit_code
#!/usr/bin/env bash
set -euo pipefail
task_id="$1"
echo "Job starting..."
sleep_sec="$((task_id % 5))"
echo 'Sleeping...'
sleep "$sleep_sec"
if [[ "$sleep_sec" = 3 ]] ; then
echo 'Something went wrong!' 1>&2
exit 1
fi
echo 'Successfully completed'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment