Bug, or documentation issue with Python's `imp` module?

.gitignore

*.pyc

__init__.py

__init__.py

README.md

####Overview

If you use import to load a package and subpackage:

import package
import package.subpackage

Then the package module instance will contain a subpackage attribute:

assert "subpackage" in dir(sys.modules['package'])

But if you use Python's imp module to import these packages instead, the same assertion will fail.

Is this a python documentation oversight, or a bug with the imp module?

####Instructions

Clone (or unzip) this repository and then:

• Run python testone.py -- all asserts pass. This tests python's basic import machinery.

• Run python testtwo.py -- asserts fail, but shouldn't. This uses the imp module instead of directly using import, but it should have the same behavior, I think?

I tested with 2.5, 2.6 and 2.7.

####Notes

Read the imp module documentation.

The tests can be reconciled as seen in reconciled.py -- but the documentation doesn't indicate this should be necessary, and it feels like imp.load_module(...) should manage the content of sys.modules and the parent module on my behalf.

PS: I found this by investigating mysterious import errors encountered when testing with NoseGAE. NoseGAE uses internal code from App Engine's dev_appserver.py infrastructure that provides import hooks via sys.meta_path. Those hooks in turn use the imp module. Ultimately, this leads to strange import errors. For example, internal Django code will fail to load Django submodules in some cases! The App Engine HardenedModulesHook class has code that looks like testtwo.py rather than reconciled.py.

reconciled.py

# 
# This file shows how you can reconcile testone.py with testtwo.py
#

import sys
import imp
assert "package" not in sys.modules

package_file, package_pathname, package_description = imp.find_module('package')
package_module = imp.load_module('package', package_file, package_pathname, package_description)
assert "package" in sys.modules
assert "subpackage" not in dir(sys.modules["package"])

subpackage_file, subpackage_pathname, subpackage_description = imp.find_module('subpackage', sys.modules["package"].__path__)
subpackage_module = imp.load_module('package.subpackage', subpackage_file, subpackage_pathname, subpackage_description)
setattr(sys.modules["package"], 'subpackage', subpackage_module)
assert "package.subpackage" in sys.modules
assert "package" in sys.modules
assert "subpackage" not in sys.modules
assert "subpackage" in dir(sys.modules["package"]), "This will pass, but perhaps it shouldn't? See testtwo.py"

try:
    from package import subpackage
except ImportError:
    assert False, "This won't be hit, but perhaps it should? See testtwo.py"
else:
    assert True

testone.py

# 
# This is the "working" test that shows the behavior of sys.modules
# when using python's import mechanism.
#

import sys
assert "package" not in sys.modules

import package
assert "package" in sys.modules
assert "subpackage" not in dir(sys.modules["package"])

import package.subpackage
assert "package.subpackage" in sys.modules
assert "package" in sys.modules
assert "subpackage" not in sys.modules
assert "subpackage" in dir(sys.modules["package"]), "This assert will fail when using the imp module instead of import."

try:
    from package import subpackage
except ImportError:
    assert False, "This assert will be hit when using the imp module instead of import."
else:
    assert True

testtwo.py

# 
# This is the "broken" test that shows the behavior of sys.modules
# when using python's imp module. This test is parallel to the test
# found in testone.py, so my expectation was that all asserts would pass.
# Is this a bug with imp, or a documentation oversight?
#

import sys
import imp
assert "package" not in sys.modules

package_file, package_pathname, package_description = imp.find_module('package')
imp.load_module('package', package_file, package_pathname, package_description)
assert "package" in sys.modules
assert "subpackage" not in dir(sys.modules["package"])

subpackage_file, subpackage_pathname, subpackage_description = imp.find_module('subpackage', sys.modules["package"].__path__)
imp.load_module('package.subpackage', subpackage_file, subpackage_pathname, subpackage_description)
assert "package.subpackage" in sys.modules
assert "package" in sys.modules
assert "subpackage" not in sys.modules
assert "subpackage" in dir(sys.modules["package"]), "This assert fails but shouldn't: imp didn't add 'subpackage' to the 'package' module"

try:
    from package import subpackage
except ImportError:
    assert False, "This assert also fails but shouldn't. To hit it, comment out the previous failing assert."
else:
    assert True