Python & LLVM Extension Modules
This is a quick proof of concept of using the LLVM JIT compiler as
Python's extension module loader. This implelentation completely
replaces the standard extension module framework, so regular extension
modules cannot be loaded. It would not be too difficult to extend the
patch to allow loading both LLVM bitcode (
.bc) and regular extension
The patch is designed to be applied over the mercurial Python 3.3 branch. It probably works for other branches, but this is untested.
hg clone http://hg.python.org/cpython hg update 3.3 cp ~/path/to/dynload_llvm.cpp Python/dynload_llvm.cpp patch -p1 < ~/path/to/python-dynload-llvm.patch
Three changes to the Python building process are requried.
We use the LLVM C++ API, so we must use a C++ linker to build the
pythonexecutable. This is accomplished with the
The LLVM extension module code is in
Python/dynload_llvm.cppso we inform the build process using
We need to know how to find the LLVM include directory and libraries. This is accomplished by setting the LLVMCONFIG variable, during the make process, to the path to
./configure --with-cxx-main DYNLOADFILE=dynload_llvm.o LLVMCONFIG=llvm-config make
Compiling extension modules is relatively simple. The loader expects
the modules to be in LLVM bitcode, which may be generated using the
-c -emit-llvm flags. The bitcode file should have the extension
.bc. For example, to compile the
xx module in the Python source:
clang -g -O2 -emit-llvm -c -I. -IInclude -o xx.bc Modules/xxmodule.c
Multiple bitcode objects may be linked together into one bitcode file using
clang -O2 -g -emit-llvm -c -I. -IInclude Modules/_math.c \ -o Modules/_math.bc clang -O2 -g -emit-llvm -c -I. -IInclude Modules/mathmodule.c \ -o Modules/mathmodule.bc llvm-link -o math.bc Modules/_math.bc Modules/mathmodule.bc
After being compiled, the extension modules may be imported and used as expected:
$ ./python Python 3.3.0+ (dynload_llvm qbase qtip tip:e606bfc02df2+, Jan 14 2013, 08:51:28) [GCC 4.6.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import xx >>> xx.__file__ './xx.bc'
Extension modules which refer to symbols in dynamically loaded libraries may be used, but the library first needs to be loaded so that its symbols are available. This can be done, for example, using ctypes:
>>> import ctypes >>> libexample = ctypes.CDLL('/path/to/libexample.so', ... ctypes.RTLD_GLOBAL) >>> import example # Now has access to symbols exported in libexample.so.
It may be necessary to add a line to
Modules/Setup to compile ctypes
into the Python module:
_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -lffi