Skip to content

Instantly share code, notes, and snippets.

@matthewfeickert

matthewfeickert/README.md

Created March 18, 2023 05:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save matthewfeickert/26e53978769c6fbe2a45bf140549f58c to your computer and use it in GitHub Desktop.
Save matthewfeickert/26e53978769c6fbe2a45bf140549f58c to your computer and use it in GitHub Desktop.
Download ZIP
Examples of problems with installPip not using virtual environments
Raw
README.md

Failure of not using a virtual environment in installPip

By not using a virtual environment installPip makes it unable to interact with the virtual environment manually as one can not reliably manually install or uninstall packages.

Example: Unable to uninstall numpy

$ setupATLAS
...
$ asetup AnalysisBase,22.2.110
Using AnalysisBase/22.2.110 [cmake] with platform x86_64-centos7-gcc11-opt
	at /cvmfs/atlas.cern.ch/repo/sw/software/22.2
Unchanged: COOL_ORA_ENABLE_ADAPTIVE_OPT=Y
$ mkdir -p /tmp/example
$ echo "numpy" > /tmp/example/requirements.txt
$ installPip $(readlink -f /tmp/example/requirements.txt)


 * * * Installing for centos7 and python 3.9 ***

Info: /cvmfs mounted; do 'setupATLAS -d -c ...' to skip default mounts.
Info: $HOME mounted; do 'setupATLAS -d -c ...' to skip default mounts.
------------------------------------------------------------------------------
Apptainer: 1.1.5
From: /cvmfs/atlas.cern.ch/repo/containers/sw/apptainer/x86_64-el7/1.1.5/bin/apptainer
ContainerType: atlas-derived
apptainer  exec  -e  -H /home/feickert/.alrb/container/apptainer/home.6nDuKc:/alrb -B /cvmfs:/cvmfs -B /home:/home -B /tmp/example/x86_64-centos7:/srv /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-alrbdev-centos7 /alrb/.bashrc
------------------------------------------------------------------------------
 setupATLAS is available for this atlas-derived container type.

    * * * Contents of /srv/userRequirements.txt   * * *
numpy
    * * * * * *
************************************************************************
Requested:  python ...
 Setting up python 3.9.14-x86_64-centos7 ...
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
TMPDIR not defined. Will use PWD=/srv

python3 -m pip install  --user virtualenv
Collecting virtualenv
  Downloading virtualenv-20.21.0-py3-none-any.whl (8.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.7/8.7 MB 132.5 MB/s eta 0:00:00
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.10.0-py3-none-any.whl (9.9 kB)
Collecting distlib<1,>=0.3.6
  Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.5/468.5 KB 112.5 MB/s eta 0:00:00
Collecting platformdirs<4,>=2.4
  Downloading platformdirs-3.1.1-py3-none-any.whl (14 kB)
Installing collected packages: distlib, platformdirs, filelock, virtualenv
  WARNING: The script virtualenv is installed in '/srv/deleteMe-pipInstalls/pipInstalls-x86_64-centos7/VirtualEnvDir/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed distlib-0.3.6 filelock-3.10.0 platformdirs-3.1.1 virtualenv-20.21.0
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.14-x86_64-centos7/bin/python3 -m pip install --upgrade pip' command.

.local/bin/virtualenv -p python3 myVenv
created virtual environment CPython3.9.14.final.0-64 in 323ms
  creator CPython3Posix(dest=/srv/deleteMe-pipInstalls/pipInstalls-x86_64-centos7/VirtualEnvDir/myVenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/srv/deleteMe-pipInstalls/pipInstalls-x86_64-centos7/VirtualEnvDir/.local/share/virtualenv)
    added seed packages: pip==23.0.1, setuptools==67.4.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

pip install  -r /srv/userRequirements.txt
Collecting numpy
  Downloading numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 126.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.24.2
pip freeze ...
numpy==1.24.2

packing numpy
OK: OS: 7, ARCH: x86_64, PYTHON: 3.9

Usage:
	source /tmp/example/setup.sh

OK: OS: 7, ARCH: x86_64, PYTHON: 3.9
$ . /tmp/example/setup.sh
************************************************************************
Requested:  python ...
 Setting up python 3.9.14-x86_64-centos7 ...
  Skipping: python 3.9 x86_64 already setup.
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
$ env | grep '/tmp/example/'
LD_LIBRARY_PATH=/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib64:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-8a51a/x86_64-centos7/lib:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-8a51a/x86_64-centos7/lib64:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/lib:/tmp/example/x86_64-centos7/lib
PATH=/tmp/feickert/.asetup-sysbin-a8pnJP_380510:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/share:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/share:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/Cmake/3.24.3/Linux-x86_64/bin:/cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-8a51a/x86_64-centos7/bin:/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin:/home/feickert/.local/bin:/home/feickert/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/opt/dell/srvadmin/bin:/tmp/example/x86_64-centos7/bin
PYTHONPATH=/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages:/tmp/example/x86_64-centos7/lib/python3.9:/tmp/example/x86_64-centos7/lib/python3.9/site-packages
$ command -v python
/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/python
$ python --version --version
Python 3.9.12 (main, Feb 23 2023, 02:25:04)
[GCC 11.2.0]
$ python -m pip list
Package    Version
---------- -------
Cython     0.29.28
LHAPDF     6.5.1
numpy      1.24.2
pip        22.0.4
setuptools 44.1.0
wheel      0.33.4
xrootd     5.4.3
$ python -m pip show numpy
Name: numpy
Version: 1.24.2
Summary: Fundamental package for array computing in Python
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD-3-Clause
Location: /tmp/example/x86_64-centos7/lib/python3.9/site-packages
Requires:
Required-by:
$ python -m pip uninstall numpy
Found existing installation: numpy 1.24.2
Uninstalling numpy-1.24.2:
  Would remove:
    /cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/f2py
    /cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/f2py3
    /cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/f2py3.9
    /tmp/example/x86_64-centos7/bin/f2py
    /tmp/example/x86_64-centos7/bin/f2py3
    /tmp/example/x86_64-centos7/bin/f2py3.9
    /tmp/example/x86_64-centos7/lib/python3.9/site-packages/numpy-1.24.2.dist-info/*
    /tmp/example/x86_64-centos7/lib/python3.9/site-packages/numpy.libs/libgfortran-040039e1.so.5.0.0
    /tmp/example/x86_64-centos7/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
    /tmp/example/x86_64-centos7/lib/python3.9/site-packages/numpy.libs/libquadmath-96973f99.so.0.0.0
    /tmp/example/x86_64-centos7/lib/python3.9/site-packages/numpy/*
Proceed (Y/n)? y
ERROR: Exception:
Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/shutil.py", line 823, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/f2py' -> '/tmp/pip-uninstall-fpapnb8y/f2py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/commands/uninstall.py", line 97, in run
    uninstall_pathset = req.uninstall(
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/req/req_install.py", line 638, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/req/req_uninstall.py", line 369, in remove
    moved.stash(path)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/req/req_uninstall.py", line 267, in stash
    renames(path, new_path)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg/pip/_internal/utils/misc.py", line 305, in renames
    shutil.move(old, new)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/shutil.py", line 844, in move
    os.unlink(src)
OSError: [Errno 30] Read-only file system: '/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.110/InstallArea/x86_64-centos7-gcc11-opt/bin/f2py'
$

There's also no deactivate script so you can't change environments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment