pool.map - multiple arguments

Example 1:  List of lists

A list of multiple arguments can be passed to a function via pool.map
(function needs to accept a list as single argument)

Example: calculate the product of each data pair

import multiprocessing
import numpy as np

data_pairs = [ [3,5], [4,3], [7,3], [1,6] ]

# define what to do with each data pair ( p=[3,5] ), example: calculate product 
def myfunc(p):
    product_of_list = np.prod(p)
    return product_of_list

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    result_list = pool.map(myfunc, data_pairs)
    print(result_list)

[15, 12, 21, 6]

Example 2: using partial()

Parallel run of a function with multiple arguments

To use pool.map for functions with multiple arguments, partial can be used to set constant values to all arguments which are not changed during parallel processing, such that only the first argument remains for iterating. (The variable input needs to be always the first argument of a function, not second or later arguments).

Example: multiply all numbers in a list by 10

import multiprocessing
from functools import partial

data_list = [1, 2, 3, 4]

def prod_xy(x,y):
    return x * y

def parallel_runs(data_list):
    pool = multiprocessing.Pool(processes=4)
    prod_x=partial(prod_xy, y=10) # prod_x has only one argument x (y is fixed to 10) 
    result_list = pool.map(prod_x, data_list) 
    print(result_list)

if __name__ == '__main__':
    parallel_runs(data_list)

[10, 20, 30, 40]


Partial creates a new simplified version of a function with part of the arguments fixed to specific values.



https://docs.python.org/3/library/multiprocessing.html
https://docs.python.org/3/library/functools.html#functools.partial