Skip to content

Instantly share code, notes, and snippets.

@tristanwietsma
Last active August 29, 2015 13:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save tristanwietsma/9961013 to your computer and use it in GitHub Desktop.
Save tristanwietsma/9961013 to your computer and use it in GitHub Desktop.
batching out jobs to separate processes with python futures
#!/usr/local/bin/python3
import sys, os
from concurrent.futures import ProcessPoolExecutor
def osjob(cmd):
print("running `{}`".format(cmd))
os.system(cmd)
print("finished `{}`".format(cmd))
if __name__ == '__main__':
f = open(sys.argv[1]) # file containing list of shell commands
jobs = [j.strip() for j in f.readlines()]
f.close()
workers = int(sys.argv[2]) # number of concurrent jobs to run
result = []
with ProcessPoolExecutor(max_workers=workers) as executor:
for job in jobs:
result.append(executor.submit(osjob, job))
for r in result:
r.result()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment