evertrol/Conda-scipy-mpi4py.txt

## Conda-scipy-mpi4py.txt
[ER; 2023-03-01]

When using the Conda channel "anaconda" and Python 3.7, the combination of the scipy package and the mpich package can cause problems. The mpich installation is a dependency of mpi4py, so the combination scipy / mpi4py has the same problem. (Note that I've tested and found the issue is the same with Python 3.10, and probably 3.8 and 3.9 as well, since mpich does not depend on Python. 3.11 is still to new (half a year old!) for the scipy package to be properly installed.)


SciPy has a dependency on libgfortran and ligfortran-ng: which one will be used, depends on the SciPy package selected, at least for libgfortran. This is the core of the problem.

When installing SciPy before MPIch, Conda will pick the most recent version. On Linux, this is scipy-1.7.3-py37h6c91a56_2 [*] (try in a new environment).
The related package file on anaconda.org is linux-64/scipy-1.7.3-py37h6c91a56_2.tar.bz2 . This shows, among others, the following dependencies:
libgfortran-ng, libgfortran5 >=11.2.0

This will thus install libgfortran5.

Linux-64-mpich in the anaconda channel does not have that many variants; see https://anaconda.org/anaconda/mpich/files (note that version 3.3.2 or 3.2.1 doesn't make a difference for this problem)

MPIch, variant mpich-3.3.2-hc856adb_0 , has this dependency: libgfortran-ng >=7,<8.0a0
Note the exclusion of version 8 and later.
libgfortran-ng >=7,<8 finds libgfortran-ng-7.5.0-h25a7a18_18 , which has one dependencie: libgfortran4 7.5.0.* .
This is libgfortran4, which of course does not match libgfortran5 installed by SciPy. Thus Conda searches for another package that matches the SciPy libgfortran5 dependency to resolve this conflict, which is the package external_0 . That package basically means (by the name "external"): I have installed MPIch myself, just install a fake package and trust me. This is why there are no gfortran or mpi dependencies: it's just a dummy package.
Unfortunately, and obviously, if you haven't installed MPIch, this won't work. Conda will install the package nevertheless, because it's the only package that matches linux-64-mpich when libgfortran5 is installed.

Ultimately, this causes MPI to fail, since the library is not available.


If you go the other way around, and install MPIch first, it will install mpich-3.3.2-hc856adb_0.
That means libgfortran4 is installed instead.
Luckily, there are many variants of SciPy in the anaconda channel. This time, Conda will skip scipy-1.7.3-py37h6c91a56_2 because of the mismatch in libgfortran4/5 dependency, and instead use scipy-1.7.3-py37hc147768_0 . This package has as fortran dependencies: libgfortran-ng, libgfortran4 >=7.5.0. These are compatible with the MPIch-installed fortran libraries, and thus this variant of SciPy will be happily installed (note: the SciPy version is still 1.7.3, even if the libgfortran version is different).


This indicates multiple ways of avoiding this issue:

- install MPIch (or mpi4py) first, then SciPy
- install them at the same time: that way, the Conda resolver will automatically notice the libgfortran4/5 issue, and resolve to using the correct library for SciPy and MPIch:
    conda install scipy mpich
  or
    conda install scipy mpi4py
  (use -c anaconda if wanted. But this is one of the default channels, so is not really necessary.)
- use an environment.yml file. This makes it also easy to store all dependencies in a single file, specify the channel(s) to use, even the environment name. Just pass the file to someone else and get the exact same environment. You should generally build the environment from scratch using this file:
  conda env create -f environment.yml


I find it hard to figure out where the actual cause of the problem is:
- the Conda dependency resolver?
- The MPIch external package (without setting dependencies)?
- Conda picking a (random-ish?) SciPy package, and not reporting that there is an issue with another package (I guess the mpich external package does not have a flag that says: please warn the user when installing this).
- something else?


[*] It's unclear to me why this particular variant of 1.7.3-py37 is installed: there are other variants, such as scipy-1.7.3-py37hc147768_0, which are more recent.
	[ER; 2023-03-01]

	When using the Conda channel "anaconda" and Python 3.7, the combination of the scipy package and the mpich package can cause problems. The mpich installation is a dependency of mpi4py, so the combination scipy / mpi4py has the same problem. (Note that I've tested and found the issue is the same with Python 3.10, and probably 3.8 and 3.9 as well, since mpich does not depend on Python. 3.11 is still to new (half a year old!) for the scipy package to be properly installed.)


	SciPy has a dependency on libgfortran and ligfortran-ng: which one will be used, depends on the SciPy package selected, at least for libgfortran. This is the core of the problem.

	When installing SciPy before MPIch, Conda will pick the most recent version. On Linux, this is scipy-1.7.3-py37h6c91a56_2 [*] (try in a new environment).
	The related package file on anaconda.org is linux-64/scipy-1.7.3-py37h6c91a56_2.tar.bz2 . This shows, among others, the following dependencies:
	libgfortran-ng, libgfortran5 >=11.2.0

	This will thus install libgfortran5.

	Linux-64-mpich in the anaconda channel does not have that many variants; see https://anaconda.org/anaconda/mpich/files (note that version 3.3.2 or 3.2.1 doesn't make a difference for this problem)

	MPIch, variant mpich-3.3.2-hc856adb_0 , has this dependency: libgfortran-ng >=7,<8.0a0
	Note the exclusion of version 8 and later.
	libgfortran-ng >=7,<8 finds libgfortran-ng-7.5.0-h25a7a18_18 , which has one dependencie: libgfortran4 7.5.0.* .
	This is libgfortran4, which of course does not match libgfortran5 installed by SciPy. Thus Conda searches for another package that matches the SciPy libgfortran5 dependency to resolve this conflict, which is the package external_0 . That package basically means (by the name "external"): I have installed MPIch myself, just install a fake package and trust me. This is why there are no gfortran or mpi dependencies: it's just a dummy package.
	Unfortunately, and obviously, if you haven't installed MPIch, this won't work. Conda will install the package nevertheless, because it's the only package that matches linux-64-mpich when libgfortran5 is installed.

	Ultimately, this causes MPI to fail, since the library is not available.


	If you go the other way around, and install MPIch first, it will install mpich-3.3.2-hc856adb_0.
	That means libgfortran4 is installed instead.
	Luckily, there are many variants of SciPy in the anaconda channel. This time, Conda will skip scipy-1.7.3-py37h6c91a56_2 because of the mismatch in libgfortran4/5 dependency, and instead use scipy-1.7.3-py37hc147768_0 . This package has as fortran dependencies: libgfortran-ng, libgfortran4 >=7.5.0. These are compatible with the MPIch-installed fortran libraries, and thus this variant of SciPy will be happily installed (note: the SciPy version is still 1.7.3, even if the libgfortran version is different).


	This indicates multiple ways of avoiding this issue:

	- install MPIch (or mpi4py) first, then SciPy
	- install them at the same time: that way, the Conda resolver will automatically notice the libgfortran4/5 issue, and resolve to using the correct library for SciPy and MPIch:
	conda install scipy mpich
	or
	conda install scipy mpi4py
	(use -c anaconda if wanted. But this is one of the default channels, so is not really necessary.)
	- use an environment.yml file. This makes it also easy to store all dependencies in a single file, specify the channel(s) to use, even the environment name. Just pass the file to someone else and get the exact same environment. You should generally build the environment from scratch using this file:
	conda env create -f environment.yml



	I find it hard to figure out where the actual cause of the problem is:
	- the Conda dependency resolver?
	- The MPIch external package (without setting dependencies)?
	- Conda picking a (random-ish?) SciPy package, and not reporting that there is an issue with another package (I guess the mpich external package does not have a flag that says: please warn the user when installing this).
	- something else?



	[*] It's unclear to me why this particular variant of 1.7.3-py37 is installed: there are other variants, such as scipy-1.7.3-py37hc147768_0, which are more recent.