Skip to content

Instantly share code, notes, and snippets.

View doublerr's full-sized avatar

Ryan Richard doublerr

  • Google
  • Austin, TX
View GitHub Profile
@mangecoeur
mangecoeur / concurrent.futures-intro.md
Last active January 9, 2024 16:04
Easy parallel python with concurrent.futures

Easy parallel python with concurrent.futures

As of version 3.3, python includes the very promising concurrent.futures module, with elegant context managers for running tasks concurrently. Thanks to the simple and consistent interface you can use both threads and processes with minimal effort.

For most CPU bound tasks - anything that is heavy number crunching - you want your program to use all the CPUs in your PC. The simplest way to get a CPU bound task to run in parallel is to use the ProcessPoolExecutor, which will create enough sub-processes to keep all your CPUs busy.

We use the context manager thusly:

with concurrent.futures.ProcessPoolExecutor() as executor: