Skip to content

Instantly share code, notes, and snippets.

@hadim
Created June 21, 2019 15:46
Show Gist options
  • Save hadim/a2e2e52b3bafad9d45d7110f17fb59cd to your computer and use it in GitHub Desktop.
Save hadim/a2e2e52b3bafad9d45d7110f17fb59cd to your computer and use it in GitHub Desktop.
# Install the joblib library if needed.
# conda install joblib
import joblib as jb
import numpy as np
def my_function_that_take_time(arg1, arg2):
# Simulate a long computation.
import time
time.sleep(5)
# Here we return the addition of arg1 and arg2
return arg1 + arg2
# Generate random list of parameters for our dummy function.
# It takes the form of a dict.
parameters_list = [{'arg1': val1, 'arg2': val2} for val1, val2 in np.random.randint(0, 50, size=(20, 2))]
# Generate th arguments for the executor.
executor_args = [jb.delayed(my_function_that_take_time)(**params) for params in parameters_list]
# Instantiate the executor.
# `n_jobs` can also be -1 to use all available CPUs
# or 1 to disable parallelization at all.
n_jobs = 4
executor = jb.Parallel(n_jobs=n_jobs)
# Execute your parallel job.
results = executor(executor_args)
# If you want to track progress
# of your computations, you can
# use a progress bar.
# (anamic is required for this part)
import anamic
executor = anamic.utils.parallel_executor(n_jobs=n_jobs)
# Execute your parallel job.
results = executor()(executor_args)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment