Skip to content

Instantly share code, notes, and snippets.

@kpittman-securus
Last active April 28, 2021 23:20
Show Gist options
  • Save kpittman-securus/9ad20bb2e0a6d73ed2429c8be6bd7f58 to your computer and use it in GitHub Desktop.
Save kpittman-securus/9ad20bb2e0a6d73ed2429c8be6bd7f58 to your computer and use it in GitHub Desktop.
Parallel execution with find and xargs
#! /usr/bin/env bash
find ./src/bookmarklets -name "*.js" -print0 | xargs -I FILE -0 -P "$(nproc)" npm run rollup -- --input "FILE" --file "output/FILE"
# EDIT: Per my latest comment - I now prefer this:
for bookmarklet in ./src/bookmarklets/**/*.js
do
npm run rollup -- --input "$bookmarklet" --file "output/$bookmarklet" &
done
wait
@kpittman-securus
Copy link
Author

-I FILE tells xargs to use FILE as a placeholder in the command... similar to the use of {} during find/--exec

-P "$(nproc)" tells xargs to run the commands in parallel, with a max of N parallel threads...
In this script I just use the result of nproc command to pass in a number (on my current system, it comes back with 8).

Then, we just run our command as normal (in this case, npm run rollup with some args).

The use of FILE placeholder will get substituted with the value that passed in from find

@kpittman-securus
Copy link
Author

I ran into a behavior of rollup that I didn't want. Specifically, it was trying to require other modules instead of inlining them.

After a few minutes of searching, I had not found a way to force inlining all code other than to disable multi-file input/output and process each file on its' own.

My first solution was to use find/--exec on each file I wanted to process.

It was taking 3+ seconds per file, which isn't a lot but also isn't super quick. So, I was able to process all files in parallel with find/xargs -P N.

Since my command consumes the filename in multiple spots, I needed to use the placeholder -I FILE.

@kpittman-securus
Copy link
Author

After some time to stew on this, and reading some alternatives, I feel that using a loop with background processes is more readable.

for thing in glob
do
  npm run doThingWithArg -- --file=thing & # & sends proc to bg
done
wait # waits for bg procs to complete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment