Some starter examples using GNU Parallel. #parallel #gnu-parallel #bash

README-gnu-parallel-examples.md

README - Examples on using GNU Parallel

Suppose we want to compress (using bzip2) a number of log files.

Distrubute jobs to several threads on local machine. The commands (for each input file) are executed in current working directory.

ls *.log| parallel -j+0 bzip2 -9 '{}' 

Distribute jobs to several SSH-reachable machines. The input files are transfered and then cleaned-up:

ls *.log | parallel
   -S localhost,jupiter.localdomain,hermes.localdomain # The SSH hosts to login and execute job
   --transfer       # Transfer (rsync) to remote computers
   --return {}.bz2  # The pattern of output file to be transfered (rsync) back after completion
   --cleanup        # Cleanup working direcory after completion
   --workdir ...    # Use a working directory (unique for a parallel execution) under ~/.parallel/tmp/ ... 
   -j+0             # Spawn N+0 threads, where N is number of available cores (for each worker machine) 
   --eta            # Print ETA progress information during execution 
   bzip2 -9 '{}'    # The actual command to be applied to each input file