Skip to content

Instantly share code, notes, and snippets.

@zchee
Forked from mtorromeo/parallelrun.py
Last active August 27, 2015 17:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zchee/c0f58bfdbfe05dfcce4a to your computer and use it in GitHub Desktop.
Save zchee/c0f58bfdbfe05dfcce4a to your computer and use it in GitHub Desktop.
Parallelize shell processes enforcing a concurrency limit using python3 and asyncio
import asyncio
concurrency_sem = asyncio.Semaphore(3)
@asyncio.coroutine
def run(process):
yield from concurrency_sem.acquire()
print("Running {}...".format(process))
proc = yield from asyncio.create_subprocess_shell(process)
yield from proc.communicate()
print("Process {} terminated.".format(process))
concurrency_sem.release()
tasks = [
asyncio.Task(run("sleep 1")),
asyncio.Task(run("sleep 2")),
asyncio.Task(run("sleep 1")),
asyncio.Task(run("sleep 2")),
asyncio.Task(run("sleep 1")),
asyncio.Task(run("sleep 2")),
asyncio.Task(run("sleep 1")),
asyncio.Task(run("sleep 2")),
]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment