Skip to content

Instantly share code, notes, and snippets.

@tdhopper
Created July 11, 2012 15:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tdhopper/3091122 to your computer and use it in GitHub Desktop.
Save tdhopper/3091122 to your computer and use it in GitHub Desktop.
Python parallelization error
from multiprocessing import Pool
from scipy.stats import uniform
def unwrap_self_f(arg, **kwarg):
return C.f(*arg, **kwarg)
class C:
def __init__(self, x = []):
self.x = x
def f(self, name):
print 'hello %s' %name
def run(self):
pool = Pool(processes=4)
names = ('frank', 'justin', 'osi', 'thomas')
pool.map(unwrap_self_f, zip([self]*len(names), names))
if __name__ == '__main__':
print "Parallelize when I only contain an integer"
c = C(x = 1)
c.run()
print "Parallelize when I contain something more complicated"
c = C(x = uniform)
c.run()
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 530, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 285, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment