20223-03-15 technical response to JIRA ticket ATLAS ASGATLASG-2477: python packages for atlasexternals
As a warning, this is going to be a long response as it is going to need to discuss the technical details of how the way LCG views and the current way ATLAS externals works is fundamentally at odds with the core concepts of Python virtual environments. To make this more digestible in a JIRA ticket thread and to avoid repeating myself (somewhat) I am going to assume that anyone who is reading this will read the README I've already written for cvmfs-venv which explains this at a high level. Also, as I find JIRA's lack of Markdown to make reading next to anything on it pretty terrible I've put the full contents of the technical parts that follow in this GitHub Gist in case you want to read it over there.
Disclaimer: Packaging and distributing software is hard. I am not blaming anyone for the current difficulties that we face and I do not know the complex history of work that has gone into getting us where were are today.
TL;DR: The way that ATLAS externals is built makes it so that the only way to use a virtual environment with it (without manually editing the virtual environment's activate scripts) is to relinquish all control of packages already in ATLAS externals, thus making some software environment configurations impossible to reach (without additional measures that I don't think are in any way reasonable to ask a user to manage themselves).
For the ease of demonstration I will be using the atlas/analysisbase:22.2.106 Docker image as an example test environment but this of course all applies to setting up the release through a CVMFS enabled machine.
$ docker run --rm -ti atlas/analysisbase:22.2.106
_ _____ _ _ ___
/_\_ _| | /_\ / __|
/ _ \| | | |__ / _ \\__ \
/_/ \_\_| |____/_/ \_\___/
This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:
source /release_setup.sh
[bash][atlas]:workdir > . /release_setup.sh
Configured GCC from: /opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/22.2.106/InstallArea/x86_64-centos7-gcc11-opt
[bash][atlas AnalysisBase-22.2.106]:workdir > command -v python # Python itself is an ATLAS external
/usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/bin/python
[bash][atlas AnalysisBase-22.2.106]:workdir > python --version --version
Python 3.9.12 (main, Jan 26 2023, 02:25:00)
[GCC 11.2.0]
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.22.3
pip 22.0.4
setuptools 44.1.0
wheel 0.33.4
xrootd 5.4.3
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip show numpy
Name: numpy
Version: 1.22.3
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg
Requires:
Required-by:
[bash][atlas AnalysisBase-22.2.106]:workdir >
Remember that when we run . /release_setup.sh anything we have access to now is fundamentally read only and outside of user space control as this would normally be software provided by CVMFS.
From the above output we can see that when we setup an AnalysisBase release the Python runtime that we have access to has a non-empty site-packages/ directory which is populated with software defined in ATLAS externals. For example purposes we will look at how NumPy has come to be installed here and the problems that this poses additional software management for user controlled virtual environments.
To begin with, we note that /release_setup.sh has set PYTHONPATH
[bash][atlas AnalysisBase-22.2.106]:workdir > echo $PYTHONPATH
/usr/AnalysisBase/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBase/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
Note that at the very end of PYTHONPATH is the site-packages/ directory that houses the NumPy that was setup by the release (/usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages)
and remember that a virtual environment does not set PYTHONPATH and so this will not change (this is the work that cvmfs-venv had to do).
To start with, we'll revisit the problem of naively trying to create a virtual environment and upgrade NumPy
$ docker run --rm -ti atlas/analysisbase:22.2.106
...
[bash][atlas]:workdir > . /release_setup.sh
...
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m venv venv && . venv/bin/activate
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.22.3
pip 22.0.4
setuptools 44.1.0
wheel 0.33.4
xrootd 5.4.3
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip install --upgrade numpy
Requirement already satisfied: numpy in /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg (1.22.3)
Collecting numpy
Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 1.3 MB/s eta 0:00:00
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Not uninstalling numpy at /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg, outside environment /workdir/venv
Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed numpy-1.24.2
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip show numpy
Name: numpy
Version: 1.22.3
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg
Requires:
Required-by:
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir >
What happened and why? The immutable read-only release of NumPy was found first because it exists on PYTHONPATH and so when the attempt to uninstall it during the upgrade fails, pip tries to continue and is able to install NumPy in the virtual environment, but the virtual environment is not on the PYTHONPATH and so will never be found at runtime given there is a NumPy on PYTHONPATH.
We can see in the following example that if we strip off this site-packages/ from PYTHONPATH while we still have access to the ATLAS externals packages if we create a Python virtual environment we lose them (defeating the purpose of externals) but gain user control over what is installed (cvmfs-venv had to do manipulation of the virtual environment's activate scripts to keep both)
$ docker run --rm -ti atlas/analysisbase:22.2.106
...
[bash][atlas]:workdir > . /release_setup.sh
...
[bash][atlas AnalysisBase-22.2.106]:workdir > export OLD_PYTHONPATH="${PYTHONPATH}"
[bash][atlas AnalysisBase-22.2.106]:workdir > export PYTHONPATH="${PYTHONPATH%:*}" # Strip off site-packages/
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m venv venv && . venv/bin/activate
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
pip 22.0.4
setuptools 58.1.0
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/workdir/venv/bin/python -m pip install --upgrade pip' command.
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip --quiet install --upgrade pip setuptools wheel
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
pip 23.0.1
setuptools 67.6.0
wheel 0.40.0
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip install --upgrade numpy
Collecting numpy
Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 1.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.24.2
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
numpy 1.24.2
pip 23.0.1
setuptools 67.6.0
wheel 0.40.0
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir >
While you can get things working with a quick hack of prepending the venv's site-packages/ directory to PYTHONPATH, you now have to manually bookkeep the PYTHONPATH if you ever decide to deactivate the virtual environment (what cvmfs-venv also has to do). Users will not do this correctly if it is left up to them.
$ docker run --rm -ti atlas/analysisbase:22.2.106
...
[bash][atlas]:workdir > . /release_setup.sh
...
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m venv venv && . venv/bin/activate
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > export PYTHONPATH=$(readlink -f $(find venv/ -type d -iname "site-packages")):"${PYTHONPATH}" # prepend venv's site-packages/
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.22.3
pip 22.0.4
setuptools 58.1.0
wheel 0.33.4
xrootd 5.4.3
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip install --upgrade numpy
Requirement already satisfied: numpy in /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg (1.22.3)
Collecting numpy
Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 1.2 MB/s eta 0:00:00
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Not uninstalling numpy at /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg, outside environment /workdir/venv
Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed numpy-1.24.2
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.24.2
pip 22.0.4
setuptools 58.1.0
wheel 0.33.4
xrootd 5.4.3
(venv) [bash][atlas AnalysisBase-22.2.106]:workdir >
Given these complication it is (sadly) no surprise that users then abandon any kind of environment definition and attempt to work with a user install as ~/.local/ is automatically picked up, and this only further reinforces behavior that inevitably leads to broken and unreproducible environments.
$ docker run --rm -ti atlas/analysisbase:22.2.106
...
[bash][atlas]:workdir > . /release_setup.sh
...
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip install --upgrade numpy
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numpy in /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg (1.22.3)
Collecting numpy
Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 1.2 MB/s eta 0:00:00
Installing collected packages: numpy
WARNING: The scripts f2py, f2py3 and f2py3.9 are installed in '/home/atlas/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.24.2
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.24.2
pip 22.0.4
setuptools 44.1.0
wheel 0.33.4
xrootd 5.4.3
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip show numpy
Name: numpy
Version: 1.24.2
Summary: Fundamental package for array computing in Python
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD-3-Clause
Location: /home/atlas/.local/lib/python3.9/site-packages
Requires:
Required-by:
[bash][atlas AnalysisBase-22.2.106]:workdir >
So to avoid all of this, have access to the ATLAS externals, and give user level environment control there would either need to be some major revision of how ATLAS externals works or we would need to implement something like cvmfs-venv (which to be clear is still an ugly hack to get out of a bad situation):
$ docker run --rm -ti atlas/analysisbase:22.2.106
...
[bash][atlas]:workdir > curl -sLO https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/3169033caf98b1e8b0a649e9c16cf8c70940a89d/atlas_setup.sh
[bash][atlas]:workdir > . atlas_setup.sh example-venv-name
Configured GCC from: /opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/22.2.106/InstallArea/x86_64-centos7-gcc11-opt
# Creating new Python virtual environment 'example-venv-name'
(example-venv-name) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.22.3
pip 23.0.1
setuptools 67.6.0
wheel 0.40.0
xrootd 5.4.3
(example-venv-name) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip install --upgrade numpy awkward
Requirement already satisfied: numpy in /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg (1.22.3)
Collecting numpy
Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 1.2 MB/s eta 0:00:00
Collecting awkward
Downloading awkward-2.1.0-py3-none-any.whl (587 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 587.6/587.6 kB 1.2 MB/s eta 0:00:00
Collecting typing-extensions>=4.1.0
Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting awkward-cpp==12
Downloading awkward_cpp-12-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 1.1 MB/s eta 0:00:00
Collecting packaging
Downloading packaging-23.0-py3-none-any.whl (42 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 kB 1.4 MB/s eta 0:00:00
Installing collected packages: typing-extensions, packaging, numpy, awkward-cpp, awkward
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Not uninstalling numpy at /usr/AnalysisBaseExternals/22.2.106/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.22.3-py3.9-linux-x86_64.egg, outside environment /workdir/example-venv-name
Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed awkward-2.1.0 awkward-cpp-12 numpy-1.24.2 packaging-23.0 typing-extensions-4.5.0
(example-venv-name) [bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list
Package Version
----------------- -------
awkward 2.1.0
awkward-cpp 12
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.24.2
packaging 23.0
pip 23.0.1
setuptools 67.6.0
typing_extensions 4.5.0
wheel 0.40.0
xrootd 5.4.3
(example-venv-name) [bash][atlas AnalysisBase-22.2.106]:workdir > deactivate
[bash][atlas AnalysisBase-22.2.106]:workdir > python -m pip list # back to original ATLAS externals
Package Version
---------- -------
Cython 0.29.28
LHAPDF 6.5.1
numpy 1.22.3
pip 22.0.4
setuptools 44.1.0
wheel 0.33.4
xrootd 5.4.3
[bash][atlas AnalysisBase-22.2.106]:workdir >
So I think Dan's concerns are very valid. If we decide to move forward with installing all the libraries of the scikit-hep metapackage then we are basically committing all users regardless of their skill level to those versions (and the versions of their dependencies) for the entire release period.