Skip to content

Instantly share code, notes, and snippets.

@ThomLecha
Last active April 18, 2024 21:28
Show Gist options
  • Save ThomLecha/cf082b44bfac14d577fe9cff56b41198 to your computer and use it in GitHub Desktop.
Save ThomLecha/cf082b44bfac14d577fe9cff56b41198 to your computer and use it in GitHub Desktop.
Easy-to-use Python module for executing functions in parallel with joblib
from joblib import Parallel, delayed
import multiprocessing
import warnings
DEDICATED_CORE_NUMBER = max(multiprocessing.cpu_count() - 2, 1)
def multiProcess(f, argumentsList, coreNumber=DEDICATED_CORE_NUMBER, prt=1):
"""Takes a function, a list of parameter sets for the function, and a number of cores to use as parameters.
Executes the function in parallel across all parameter sets (using 'coreNumber' cores)
Returns the list of function returns.
EXAMPLE:
def f(a, n=10):
r = 0
for i in range(n):
r += a
return (r, str(a) + str(n))
results = mp.multiProcess(f, [(3, 5), (3, 10), (3, 15)])
results will be [(15, '35'), (30, '310'), (45, '315')]
"""
if len(argumentsList) > DEDICATED_CORE_NUMBER:
txt = "Warning: "
txt += "The program can only use " + str(DEDICATED_CORE_NUMBER) + " CPU cores,"
txt += "\n however, it has been requested to execute the function " + str(
len(argumentsList)) + " times simultaneously."
warnings.warn(txt)
if prt == 1:
print("Using " + str(len(argumentsList)) + " cores out of " + str(
multiprocessing.cpu_count()) + " on the processor.")
resultsList = Parallel(n_jobs=coreNumber)(delayed(f)(*arguments) for arguments in argumentsList)
return (resultsList)
# EXAMPLE OF USE
def f(a, n=10):
r = 0
for i in range(n):
r += a
return (r, str(a) + str(n))
results = multiProcess(f, [(3, 5), (3, 10), (3, 15)])
print(results)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment