Created
September 14, 2016 12:22
-
-
Save HakimCassimallyBBC/e6d4d88dd20b03eaf2514124505d78a4 to your computer and use it in GitHub Desktop.
multicurl attempt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
FIFO=$(mktemp /tmp/multicurl.XXXXX) | |
echo FIFO=$FIFO | |
for arg in "$@" | |
do | |
curl $arg > $FIFO & | |
done | |
# this doesn't work as: | |
# - both are noisy | |
# - both streams overlap (want line buffered) | |
# - ctrl-c leaves the curls running in background |
# Assuming bash
pids=()
for arg in "$@"; do
curl | while read -r line; do printf "%s\n" "$line" >"$FIFO"; done &
pids+="$!"
done
trap 'for pid in "${PIDS}"; do kill $pid; done' EXIT
while true; do cat "$FIFO"; done
# easier in Perl probably?
use strict;
use warnings;
my @FH = map {
open (my $FH, '-|', 'curl', '-s', '-k', $_);
$FH;
} @ARGV;
while (1) {
my @lines = grep defined, map scalar <$_>, @FH or last;
print @lines;
}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To avoid interleaving you might need something like:
Terminating when all the curl processes have finished is not solved.