Skip to content

Instantly share code, notes, and snippets.

@gavvvr
Last active September 11, 2020 00:04
Show Gist options
  • Save gavvvr/2529a6ae313dfed60138c3890e78998c to your computer and use it in GitHub Desktop.
Save gavvvr/2529a6ae313dfed60138c3890e78998c to your computer and use it in GitHub Desktop.
Running multiple commands in parallel on unix shells with `xargs`
echo 'basiclite sdk sqlplus' | tr ' ' '\n' | xargs -P 3 -I{} \
sh -c "wget -O {}.zip \
https://download.oracle.com/otn_software/linux/instantclient/19800/instantclient-{}-linux.x64-19.8.0.0.0dbru.zip \
&& unzip {}.zip -d $CLIENT_PATH && rm {}.zip"

I used this command to add a content of multiple archives from the Internet within single RUN step in Dockerfile which will lead to only one new layer.

Explanation

My goal was to perform the set of same operations (download, unpack and remove) but for 3 different files. Can I do it in DRY-manner? xargs to the rescure. If we only had one command to execute for each space-separated input, the xargs usage could be that simple:

echo 'basiclite sdk sqlplus' | xargs -n1 echo

which would execute the xargs command producing the following output:

basiclite
sdk
sqlplus

To reuse argument in arbitrary places multiple times -I{} is used (where {} is an arbitrary placeholder which can be any). Since I have multiple shell command involved, they get executed using sh -c "...". If you use -I, -n1 will not work anymore treating your input always as a single string. Using -d ' ' to split input at space char will also lead to some undesired behavior

The solution is to turn space-saparated words into multiple strings by using tr

Ah, and all of this can be executed in parallel (-P 3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment