This is incomplete; please consider contributing to the documentation effort.
In order for Hy and Python to work together as nicely as it does, Hy code needs to be able to import Python code, and vice versa.
This is done through Python’s import hooks. However, since the implementation and feature set available differs between the various versions of Python, its worth noting how the system works and the various limitations and quirks, so not to fall into certain pitfalls.
- State “DONE” from “TODO” [2018-03-31 Sat 22:45]
Python new import hooks, as specified in PEP 302, allows Hy to integrate with Python seamlessly, with Hy code able to import Python code and Python code able to import Hy code.
The new import hooks allows Hy to hook into the Python import hooks and customize them. When a Hy module is requested, Hy has a chance to look for its own modules, evaluate them, and load them into Python.
The implementation of the Python import system differ greatly between Python 3 and Python 2, and thus impacts Hy’s ability to integrate into the wider Python ecosystem.
Python 3’s import system is backed importlib
and most of its
import mechanism is pure Python code.
The Python 2 import system, in contrast, is backed by the deprecated
and lower level imp
library and a mix of exposed Python code and
built-in functionality.
https://docs.python.org/3/reference/import.html#finders-and-loaders
If the named module is not found in
sys.modules
, then Python’s import protocol is invoked to find and load the module. This protocol consists of two conceptual objects,finders
andloaders
. A finder’s job is to determine whether it can find the named module using whatever strategy it knows about. Objects that implement both of these interfaces are referred to as importers - they return themselves when they find that they can load the requested module.
Python always first tries to find a matching entry inside
sys.modules
, and if one is found, returns that.
Should the module not be found, then there are slight differences between the two platforms.
On Python 3, the process starts by looking at sys.meta_path
. This
variable contains an array of module loaders. Python tries them,
one at a time, until a loader returns successfully loads a
module. Typically contains three default entries:
- The
BuiltinImporter
which handles finding builtin modules, such asbuiltins
orsys
. - The
FrozenImporter
which handles finding frozen modules. - And the
PathFinder
which handles finding Python modules in the filesystem.
There may be a few more entries, depending on the setup, as
packages like six
and pkg_resources
hook into this mechanism to
implement some of their functionality. For example, six
hooks
into the import system to make Python 3 import paths work on
Python 2.
Hy injects a Hy module finder into the sys.meta_path
array right
before the builtin PathFinder
. The reasons for why Hy code must
be attempted first is elaborated on below.
Finders do not actually load modules. If they can find the named module, they return a module spec, an encapsulation of the module’s import-related information, which the import machinery then uses when loading the module.
The module finder then is split into three components:
- The
HyPathFinder
- The
HyFileFinder
- And the
HyLoader
The three parts provide a ModuleSpec
which Python is able to then
convert into a module.
Python 2 also goes through the sys.meta_path
list as the first
step of its import process, this time calling find_module
.
By default on Python 2, this list is empty, but Hy will hook itself into the interpreter by injecting an entry into this list.
- The Hy metaloader must be specified first, and can’t be fallen back on
- Valid Hy modules could be confused as valid Python namespace modules. This means that a Hy module could correctly import, but not contain any attributes.
- We can’t support namespace modules because otherwise valid Python modules start looking like Hy namespace modules (reverse of the problem above).
While its generally not a good idea to mix Hy and Python code in the same package, as it can lead to confusing behaviours, it technically works under Python 3, but not under Python 2.
I modified a tiny bit the code to run it :
gives me :
Python3 (original importlib)
I guess that is what you would expect should happen ?
So I seem to have a different behaviour than you describe (at least on my python 3.5).
Here are few remarks :
This likely comes from reusing SourceFileLoader without modification. overriding
find_spec
should be enough to get rid of that problem. Not sure if that is intended or not however. I remember having that problem before (maybe when I was taking python3 importlib and putting it in python2, which won't work, since the API used is different, with different behaviour expectations - think PEP451 requires python 3.4+ ), but I didn't experience that problem on my python 3.5 on that code example, using python's importlib.I would advise you to really dig into filefinder2. There are quite a few tests to validate behaviour on different python versions.
As a side note, if I run that code on python2 with filefinder2, with a simple :
at the beginning, I get :
which is more or less the result you described I believe.
Here is a quickly written class to fix it :
which should give you the python3 behaviour (minus namespace package, that becomes a built-in... there might still be some tuning to do there...) :
Adding this specialized badbyFileFinder should also be enough to fix your use case, since filefinder2 is mostly a copy of importib2, which is a mostly a copy python3 importlib code anyway.
So looking at all this, it seems that we could even make filefinder2 implementation better, by modifying a little the filefinder2.FileFinder class, so that the specialized class is not needed... We would need to be very careful to not break anything else however (hence the tests that are already there - any change comes with a lot of surprises...).
Thinking about this, I believe I do not test using the FileFinder/PathFinder class as an interface yet, I only test that various
import
calls behave in consistent ways across all python's versions, where possible... PR very welcome ;-) it would be very good to have, at least as an example of what should work and what shouldn't.