As a user of scientific computing software packages, Python packages always trips over binary extensions that require cooperation with binaries on the target system. Packaging things in wheels becomes impossible, because there is currently no way for a user to specify that they want a wheel, but specifically the wheel that is compatible with their local library.
Take mpi4py
, the most popular parallelization scheme in my field. It requires to know which MPI
implementation is used: MPICH
,
OpenMPI
, ...? What mpi4py
does is provide a source distribution that detects the local library at install time, and errors out if it can't. So the user is
tasked with installing the binary dependencies of a Python package, before they can install the Python package. Python users have by design 0 knowledge of building binary software, we pip install
or riot ┬─┬ノ(ಠ_ಠノ).
To include the binary dependency, the maintainer of the package should be able to provide several choices to the installing user.
- Do you want me to install a bundled
MPI
implementation for you? - Or, would you like me to detect a local library you already have, and build against that?
User's response:
- I don't have MPI, don't know how to build it, please take the wheel bundled with
OpenMPI
:pip install mpi4py{mpi:openmpi}
- I already have MPI, and am aware that
mpi4py
(or was made aware bympi4py
's docs) that it needs to cooperate with my local library:pip install mpi4py{mpi:local}
Same story for h5py
: pip install h5py{hdf5:local}
/pip install h5py{hdf5:bundled}
/pip install h5py{hdf5:bundled, parallel:true}
Same for any neuroscience simulator that requires MPI
, gpu, ...:
pip install arbor{mpi:local, gpu:nvidia}
,pip install NEURON{mpi:local}
,pip install nest{mpi:local}
Here's the CUDA thread I was talking about: https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125
Also, these are basically about your proposal:
https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458
https://discuss.python.org/t/archspec-a-library-for-labeling-optimized-binaries/3149
https://discuss.python.org/t/idea-selector-packages/4463
Looks like no development though.