As of version 3.3, python includes the very promising concurrent.futures module, with elegant context managers for running tasks concurrently. Thanks to the simple and consistent interface you can use both threads and processes with minimal effort.
For most CPU bound tasks - anything that is heavy number crunching - you want your program to use all the CPUs in your PC. The simplest way to get a CPU bound task to run in parallel is to use the ProcessPoolExecutor, which will create enough sub-processes to keep all your CPUs busy.
We use the context manager thusly:
with concurrent.futures.ProcessPoolExecutor() as executor:
result = executor.map(function, iterable)The snippet above uses the map() function which is by far the easiest way to split up a parallel taks. It simply appies the function provided to each element in iterable. The function could be complex and the iterable could be a list of large data tables - or even the same table, for instance, if you want to run a function 10 times on the same data, you could do:
executor.map(fun, [data] * 10)
I’ve found this is one of the easiest and most reliable approaches. Functionally it appears to be equivalent to using the multiprocessing module in this way:
pool = multiprocessing.Pool()
pool.map(…)However when using this in combination with pandas I have run into odd, hard to track down bugs where pool.map sometimes refused to work with an iterator longer than the number of processes - even though other times it worked fine. I haven’t yet had this problem with concurrent.futures and I suspect it’s a result of some strange interaction between pandas data and the internal mechanisms used to pass python objects between processes.
By using the ProcessPoolExecutor we are creating new processes and so circumventing the GIL (Global Interpreter Lock). We wont go into the GIL just now - basically it means you can't use Threads for CPU intensive parallel processing in the way you might in other languages
You can use Threads via the ThreadPoolExecutor, like so:
with concurrent.futures.ThreadPoolExecutor() as executor:
result = executor.map(function, iterable)You just change the class name and you are good to go. Using Threads you run into the GIL - your threads won’t run truely in parallel. However if you are doing IO-bound work (like reading in files or accessing the network) these usually release the GIL, so in fact you will see a performance improvement. For example say you have to make 1000 network requests, using concurrent.futures will make it so you don’t get stuck waiting for a few slow network responses when you could be running all the other ones. Because you just have to change one bit of code, you can easily experiment to see what works.