Dynamic module reload from given site package

installations.sh

# This script contains instructions that can be used to showcase how the reloader.py script is suppsoed to be used.
# The module reloading itself cotnains only pure python, so the only thing you'll need to run it is a working
# python3 interpreter

# setup start:
$ python3 -m venv env_0_12  # version 0.12 is going to be installed in here ...
$ python3 -m venv env_0_13  # ... and version 0.13 in here
$ env_0_12/bin/python3 -m pip install pyarrow==0.12
[...]
Successfully installed numpy-1.16.4 pyarrow-0.12.0 six-1.12.0
$ env_0_13/bin/python3 -m pip install pyarrow==0.13
[...]
Successfully installed numpy-1.16.4 pyarrow-0.13.0 six-1.12.0
# setup done.

# The reloader utlity can be tested by running it with any of the venvs iterpreters, e.g.
$ env_0_12/bin/python3 reloader.py
0.12.0
0.13.0
# it works! 🎉

# Running it with an unrelated interpreter that has a different version has lead to a range
# of bugs in my case, probable because numpy is opinionated about python versions. This
# implies that the whole "reloading a different version of module x" might fail horribly if
# their dependencies were updated, or if they were installed with different python versions.

reloader.py

import sys
from importlib import reload, import_module
from types import ModuleType


def deep_reload(parent, module, seen):
    name = module.__name__
    if not name.startswith(parent) or name in seen:
        # don't care about unrelated modules, avoid infinite loops
        return
    seen.add(name)
    try:
        reload(module)
    except AttributeError:
        # happens if different versions define different submodules
        return
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)
        if type(attribute) is ModuleType:
            deep_reload(parent, attribute, seen)


class TempSysPath:
    def __init__(self, path):
        self.path = path
        self.old_path = sys.path

    def __enter__(self):
        sys.path = [self.path] + sys.path

    def __exit__(self, exc_type, exc_val, exc_tb):
        sys.path = self.old_path


def load_module(module, path):
    """Load a module (and all of its defined sub-modules) given a path.

    This will load a module independent of the running python interpreter's
    site package directory.

    Args:
        module (str): Name of the module you want to load/reload.
        path (str): Absolute path to the site package where the module is
            installed.

    Returns:
        ModuleType: The module that was loaded. It can also be imported at any
            other location by just writing 'import module', this function
            doesn't do anything special after loading it into sys.modules.

    """
    with TempSysPath(path):
       if module not in sys.modules:
           import_module(module)
       else:
           # handling module references within the module itself is hard, so
           # just reload twice to be on the safe side
           deep_reload(module, sys.modules[module], set())
           deep_reload(module, sys.modules[module], set())
    return sys.modules[module]


if __name__ == '__main__':
    # load version 0.12
    print(load_module(
        'pyarrow',
        '/path/to/env_0_12/lib/python3.6/site-packages').__version__)

    # load version 0.13
    print(load_module(
        'pyarrow',
        '/path/to/env_0_13/lib/python3.6/site-packages').__version__)
 

Credit for the idea for recursive module reload goes to matthew: https://stackoverflow.com/a/17194836/962190, but their version had a couple of bugs if unrelated modules were reloaded, or they referred each other in infinite loops.