Skip to content

Instantly share code, notes, and snippets.

@adamloving adamloving/
Last active Jun 19, 2018

What would you like to do?
parallelize python method
import multiprocessing, concurrent.futures
from functools import partial
WORKER_THREAD_COUNT = multiprocessing.cpu_count()
def parallelize(partials):
results = []
with concurrent.futures.ProcessPoolExecutor(max_workers=WORKER_THREAD_COUNT) as executor:
jobs = [ executor.submit(p) for p in partials ]
for future in concurrent.futures.as_completed(jobs):
return results
# example
def work(x): print(x)
items = [1,2,3]
partials = map(lambda item: partial(work, item), items)
results = parallelize(partials)

This comment has been minimized.

Copy link

commented Jun 19, 2018



This comment has been minimized.

Copy link
Owner Author

commented Jun 19, 2018

Refactored using partials (curried functions) so that parallelize doesn't need to care about function arguments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.