Skip to content

Instantly share code, notes, and snippets.

@adamloving adamloving/parallelize.py
Last active Jun 19, 2018

Embed
What would you like to do?
parallelize python method
import multiprocessing, concurrent.futures
from functools import partial
WORKER_THREAD_COUNT = multiprocessing.cpu_count()
def parallelize(partials):
results = []
with concurrent.futures.ProcessPoolExecutor(max_workers=WORKER_THREAD_COUNT) as executor:
jobs = [ executor.submit(p) for p in partials ]
for future in concurrent.futures.as_completed(jobs):
results.append(future.result())
return results
# example
def work(x): print(x)
items = [1,2,3]
partials = map(lambda item: partial(work, item), items)
results = parallelize(partials)
@davepeck

This comment has been minimized.

Copy link

commented Jun 19, 2018

👍

@adamloving

This comment has been minimized.

Copy link
Owner Author

commented Jun 19, 2018

Refactored using partials (curried functions) so that parallelize doesn't need to care about function arguments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.