Skip to content

Instantly share code, notes, and snippets.

@samuelstjean
Last active July 18, 2020 19:41
Show Gist options
  • Save samuelstjean/7286b3377e448b8ca7370bc6dc628fd5 to your computer and use it in GitHub Desktop.
Save samuelstjean/7286b3377e448b8ca7370bc6dc628fd5 to your computer and use it in GitHub Desktop.
loky and pyinstaller test issues
#!/usr/bin/env python
import argparse
import numpy as np
from multiprocessing import freeze_support
from joblib import Parallel, delayed, parallel_backend
# this works
backend1 = 'threading'
backend2 = 'multiprocessing'
# This doesn't work
backend3 = 'loky'
backends = [backend1, backend2, backend3]
def buildArgsParser():
p = argparse.ArgumentParser()
#####################
# Required arguments
#####################
required = p.add_argument_group('Required arguments')
required.add_argument('input', metavar='input',
help='Path of the image file to denoise.')
required.add_argument('output', metavar='output',
help='Path for the saved denoised file.')
return p
def main():
parser = buildArgsParser()
args = parser.parse_args()
out = estimate_from_dwis()
def estimate_from_dwis( ncores=-1, verbose=0):
data = np.ones(100)
ranger = range(100)
print('Entering inner loop')
for backend in backends:
with parallel_backend(backend):
output = Parallel(n_jobs=ncores, verbose=verbose)(delayed(_inner)(data[i]) for i in ranger)
print('Inner loop finished for backend {}'.format(backend))
def _inner(data):
return data
if __name__ == "__main__":
freeze_support()
main()
@samuelstjean
Copy link
Author

samuelstjean commented Mar 10, 2020

When running with python test.py aa aa, works as expected. When frozen with pyinstaller, it triggers joblib/loky#236
Example (needs pyinstaller and not latest setuptools somehow):

Tested on ubuntu 16.04 with pyinstaller 3.6 and python 3.7 from conda

pip install setuptools==44 pyinstaller joblib numpy
pyinstaller test.py --onefile
./dist/test aa aa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment