Skip to content

Instantly share code, notes, and snippets.

@btel
Last active June 18, 2021 15:59
Show Gist options
  • Save btel/ec53217c41f965ffd6080a2632e9f944 to your computer and use it in GitHub Desktop.
Save btel/ec53217c41f965ffd6080a2632e9f944 to your computer and use it in GitHub Desktop.
testing mutliprocessing pool with path hooks

sys.path_hooks are not respected by multiprocessing when start mode is 'spawn' (default on Windows). However this works on Linux:

To test:

git clone https://gist.github.com/ec53217c41f965ffd6080a2632e9f944.git
cd ec53217c41f965ffd6080a2632e9f944
python test_pathtomodule.py
import sys
from multiprocessing import Pool
import pickle
import testmodule
class Loader:
def load_module(self, full_path):
if full_path == 'test':
if 'test' not in sys.modules:
sys.modules['test'] = testmodule
return testmodule
class Finder:
def __init__(self, path):
if path != 'placeholderpath':
raise ImportError()
def find_module(self, fullname, path=None):
if fullname != 'test':
return
return Loader()
sys.path_hooks.append(Finder)
sys.path.insert(0, 'placeholderpath')
from test import MyClass
if __name__ == '__main__':
pool = Pool(3)
cl = MyClass()
cl.myfunc(3)
m = pool.map(cl.myfunc, range(3))
print(list(m))
pool.terminate()
class MyClass:
def myfunc(self, i):
return i + 4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment