Skip to content

Instantly share code, notes, and snippets.

@DougBurke

DougBurke/README.md

Last active December 9, 2019 13:26
Show Gist options
  • Save DougBurke/9c13626a5a03e7fae68531e54b6c8619 to your computer and use it in GitHub Desktop.
Save DougBurke/9c13626a5a03e7fae68531e54b6c8619 to your computer and use it in GitHub Desktop.
Download ZIP
An attempt to document my problems (well, one of my problems) described at https://twitter.com/doug_burke/status/1203678875466784768
Raw
README.md

I would like to create a Python module which provides an interface to C++ code (in my actual use case the C++ function I want to access follows an API, and I use routines/include files provided with another Python module to create the C++-to-Python API (so the code in wrapit.cxx in this example).

The fun comes because the C++ code - here modelled so effortlessly by foo.cxx - relies on some FORTRAN code (bar.f in our little recreation play).

So, I want the FORTRAN code to be compiled and linked into the final library, but I do NOT want a Python interface to the FORTRAN code. I only want to call routines defined in the C++ code.

The setup.py included here does not do this (unsurprisingly really, otherwise why go to all this effort :-). Instead, f2py seems to be creating a Python module that interfaces with the FORTRAN code, and this conflicts with my wonderful hand-generated module code (wrapit.cxx). Here's the output from python setup.py build showing this conflict:

%  python setup.py build
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "wrap" sources
f2py options: []
f2py:> build/src.linux-x86_64-3.7/wrapmodule.c
creating build
creating build/src.linux-x86_64-3.7
Reading fortran codes...
	Reading file 'bar.f' (format:fix,strict)
Post-processing...
	Block: wrap
			Block: ftw
Post-processing (stage 2)...
Building modules...
	Building module "wrap"...
		Creating wrapper for Fortran function "ftw"("ftw")...
		Constructing wrapper function "ftw"...
		  ftw = ftw(npts)
	Wrote C/API module "wrap" to file "build/src.linux-x86_64-3.7/wrapmodule.c"
	Fortran 77 wrappers are saved to "build/src.linux-x86_64-3.7/wrap-f2pywrappers.f"
  adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.c' to sources.
  adding 'build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7' to include_dirs.
creating build/src.linux-x86_64-3.7/build
creating build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7
copying /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/f2py/src/fortranobject.c -> build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7
copying /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/f2py/src/fortranobject.h -> build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7
  adding 'build/src.linux-x86_64-3.7/wrap-f2pywrappers.f' to sources.
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'wrap' extension
compiling C sources
C compiler: gcc -pthread -B /home/dburke/anaconda3/envs/ciao412-usermodels/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-3.7/build
creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/build
creating build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7
compile options: '-Ibuild/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7 -I/home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include -I/home/dburke/anaconda3/envs/ciao412-usermodels/include/python3.7m -c'
gcc: build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.c
gcc: build/src.linux-x86_64-3.7/wrapmodule.c
In file included from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832:0,
                 from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.h:13,
                 from build/src.linux-x86_64-3.7/wrapmodule.c:15:
/home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832:0,
                 from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.h:13,
                 from build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.c:2:
/home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
compiling C++ sources
C compiler: g++ -pthread -B /home/dburke/anaconda3/envs/ciao412-usermodels/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC

compile options: '-Ibuild/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7 -I/home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include -I/home/dburke/anaconda3/envs/ciao412-usermodels/include/python3.7m -c'
g++: wrapit.cxx
g++: foo.cxx
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-Ibuild/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7 -I/home/dburke/anaconda3/envs/ciao412-usermodels/lib/python3.7/site-packages/numpy/core/include -I/home/dburke/anaconda3/envs/ciao412-usermodels/include/python3.7m -c'
gfortran:f77: bar.f
gfortran:f77: build/src.linux-x86_64-3.7/wrap-f2pywrappers.f
creating build/lib.linux-x86_64-3.7
g++ -pthread -shared -B /home/dburke/anaconda3/envs/ciao412-usermodels/compiler_compat -L/home/dburke/anaconda3/envs/ciao412-usermodels/lib -Wl,-rpath=/home/dburke/anaconda3/envs/ciao412-usermodels/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/wrapmodule.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.o build/temp.linux-x86_64-3.7/wrapit.o build/temp.linux-x86_64-3.7/foo.o build/temp.linux-x86_64-3.7/bar.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/wrap-f2pywrappers.o -o build/lib.linux-x86_64-3.7/wrap.cpython-37m-x86_64-linux-gnu.so
/home/dburke/anaconda3/envs/ciao412-usermodels/compiler_compat/ld: build/temp.linux-x86_64-3.7/wrapit.o: in function `PyInit_wrap':
/home/dburke/sherpa/xspec-models/dpooley/example/wrapit.cxx:25: multiple definition of `PyInit_wrap'; build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/wrapmodule.o:/home/dburke/sherpa/xspec-models/dpooley/example/build/src.linux-x86_64-3.7/wrapmodule.c:287: first defined here
collect2: error: ld returned 1 exit status
error: Command "g++ -pthread -shared -B /home/dburke/anaconda3/envs/ciao412-usermodels/compiler_compat -L/home/dburke/anaconda3/envs/ciao412-usermodels/lib -Wl,-rpath=/home/dburke/anaconda3/envs/ciao412-usermodels/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/wrapmodule.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/build/src.linux-x86_64-3.7/fortranobject.o build/temp.linux-x86_64-3.7/wrapit.o build/temp.linux-x86_64-3.7/foo.o build/temp.linux-x86_64-3.7/bar.o build/temp.linux-x86_64-3.7/build/src.linux-x86_64-3.7/wrap-f2pywrappers.o -o build/lib.linux-x86_64-3.7/wrap.cpython-37m-x86_64-linux-gnu.so" failed with exit status 1
Raw
bar.f
FUNCTION FTW(NPTS)
IMPLICIT NONE
INTEGER NPTS
INTEGER FTW
FTW = NPTS + 1
RETURN
END FUNCTION
Raw
foo.cxx
#include <iostream>
extern "C" {
// highly-complex FORTRAN
int ftw_(int n);
// highly-complex C++
void cxxtw(const int n) {
std::cout << "Input was : [" << n << "]" << std::endl;
int out = ftw_(n);
std::cout << "Output was: [" << out << "]" << std::endl;
}
}
Raw
setup.py
import setuptools
from numpy.distutils.core import setup, Extension
mod = Extension('wrap',
# include_dirs=includes,
# library_dirs=libs,
# libraries=libnames,
sources=['wrapit.cxx', 'foo.cxx', 'bar.f'])
setup(name='wrap',
version='1.0',
description='The most awseome extension never-to-have-been compiled(tm)',
ext_modules=[mod])
Raw
wrapit.cxx
// Python inteface to bar::FTW via foo::CXXTW
//
//
#include <Python.h>
extern "C" {
int ftw_(int n);
void cxxtw(const int n);
}
// Need a routine to convert from Python to C++ - it's not the focus of this
// example so this is not correct(tm).
void stub(void) {
// complex code goes here
cxxtw(20);
}
static PyMethodDef Wrappers[] = {
{ "allthethings", (PyCFunction) stub, METH_VARARGS, "Do all the things"},
{ NULL, NULL, 0, NULL }
};
static struct PyModuleDef wrap = {
PyModuleDef_HEAD_INIT,
"wrap",
NULL,
-1,
Wrappers,
};
PyMODINIT_FUNC PyInit_wrap(void) {
return PyModule_Create(&wrap);
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment