Skip to content

Instantly share code, notes, and snippets.

@0atman
Last active February 5, 2020 10:02
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save 0atman/31d85fbad4e1cf68b218267c8b57d5ae to your computer and use it in GitHub Desktop.
Save 0atman/31d85fbad4e1cf68b218267c8b57d5ae to your computer and use it in GitHub Desktop.
A process-based pmap with sensible defaults, ready to go
from multiprocessing import Pool
def pmap(f, collection, size=10):
"""
Applies `f` in parallel over `collection`.
Pool `size` has a sensible default of 10.
"""
with Pool(size) as p:
return p.map(f, collection)
@0atman
Copy link
Author

0atman commented Nov 4, 2019

And the Threaded version, useful on AWS Lamda, where processes are restricted:

from concurrent.futures import ThreadPoolExecutor

def tmap(f, collection):
    """
    Applies `f` in parallel threads over `collection`.
    Pool `size` has a sensible default of 10.
    """
    with ThreadPoolExecutor(max_workers=10) as executor:
        return list(executor.map(f, collection))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment