Skip to content

Instantly share code, notes, and snippets.

@0atman 0atman/pmap.py
Last active Feb 5, 2020

Embed
What would you like to do?
A process-based pmap with sensible defaults, ready to go
from multiprocessing import Pool
def pmap(f, collection, size=10):
"""
Applies `f` in parallel over `collection`.
Pool `size` has a sensible default of 10.
"""
with Pool(size) as p:
return p.map(f, collection)
@0atman

This comment has been minimized.

Copy link
Owner Author

0atman commented Nov 4, 2019

And the Threaded version, useful on AWS Lamda, where processes are restricted:

from concurrent.futures import ThreadPoolExecutor

def tmap(f, collection):
    """
    Applies `f` in parallel threads over `collection`.
    Pool `size` has a sensible default of 10.
    """
    with ThreadPoolExecutor(max_workers=10) as executor:
        return list(executor.map(f, collection))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.