matthewfeickert/README.md

## README.md

      
    Raw
  

              README.md
            
          
    Bootstrap example

Bootstrap example to get around bugs with panda and rucio on CVMFS and PYTHONPATH pollution as described in the March 2023 PATHelp email thread "ROOT undefined symbol error when setting up ROOT and Rucio together after setupATLAS".
Example

The PYTHONPATH is polluted with Python 3.6, so make an environment with the last pip and pandas releases that were compatible with Python 3.6.
The boostrap.sh Bash script sets up a Python virtual environment that has a version of pip that won't break with Python 3.6 which is further used in example.sh.
After the first time the environment has been setup it can be setup without the bootstrap step by sourcing example.sh again.
$ ssh <somewhere with CVMFS>
# $ git clone https://gist.github.com/b3c1815b06a972582716ef358619b982.git bootstrap-example  # if you don't have SSH keys setup
$ git clone git@gist.github.com:b3c1815b06a972582716ef358619b982.git bootstrap-example  # if you do have SSH keys
$ cd bootstrap-example
$ . example.sh

relevant output:
...
************************************************************************
Requested:  rucio xrootd panda pyami ...
 Setting up python 3.9.14-x86_64-centos7 ...
 Setting up emi 23.02.22-snapshot-x86_64-centos7 ...
 Setting up rucio 1.30.3 ...
Info: Setting compatibility to centos7
Info: Set RUCIO_AUTH_TYPE to x509_proxy
 Setting up xrootd 5.5.3-x86_64-centos7 ...
 Setting up panda 1.5.40 ...
 Setting up pyami pyAMI-5.1.3 ...
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
 emi:
   Your proxy has 10h:28m:55s remaining
 pyami:
   Using voms proxy for authentication.
************************************************************************
...
Python 3.9.14 (main, Sep 21 2022, 20:28:08)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
PYTHONPATH=/home/feickert/bootstrap-example/romain-example/lib/python3.9/site-packages:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/pyAmi/pyAMI-5.1.3/lib:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/PandaClient/1.5.40/lib/python3.6/site-packages:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/xrootd/5.5.3-x86_64-centos7/lib/python3.9/site-packages:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.30.3/lib/python3.6/site-packages:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/23.02.22-snapshot-x86_64-centos7/usr/lib64/python3.6/site-packages:/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/23.02.22-snapshot-x86_64-centos7/usr/lib/python3.6/site-packages
# python -m pip list
Package            Version
------------------ ---------
argcomplete        1.12.3
attrs              22.2.0
awkward            2.1.1
awkward-cpp        12
certifi            2022.12.7
charset-normalizer 2.0.12
dataclasses        0.8
decorator          5.1.1
dogpile.cache      1.1.5
gfal2_util         1.8.0
idna               3.4
importlib-metadata 4.8.3
jsonschema         3.2.0
numpy              1.24.2
packaging          23.0
panda-client       1.5.40
pandas             1.1.5
pbr                5.11.1
pip                21.3.1
python-dateutil    2.8.2
pytz               2022.7.1
requests           2.27.1
rucio-clients      1.30.3
setuptools         67.6.0
six                1.16.0
stevedore          3.5.2
tabulate           0.8.10
typing_extensions  4.1.1
uproot             5.0.5
urllib3            1.26.8
wheel              0.40.0
xrootd             0.0.0
zipp               3.6.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/feickert/bootstrap-example/romain-example/bin/python -m pip install --upgrade pip' command.
# python -m pip list --local
Package         Version
--------------- --------
awkward         2.1.1
awkward-cpp     12
numpy           1.24.2
packaging       23.0
pandas          1.1.5
pip             21.3.1
python-dateutil 2.8.2
pytz            2022.7.1
setuptools      67.6.0
uproot          5.0.5
wheel           0.40.0
WARNING: You are using pip version 21.3.1; however, version 23.0.1 is available.
You should consider upgrading via the '/home/feickert/bootstrap-example/romain-example/bin/python -m pip install --upgrade pip' command.
command -v rucio: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.30.3/bin/rucio
command -v xrootd: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/xrootd/5.5.3-x86_64-centos7/bin/xrootd
command -v prun: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/PandaClient/1.5.40/bin/prun
command -v ami: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/pyAmi/pyAMI-5.1.3/bin/ami
<module 'rucio' from '/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.30.3/lib/python3.6/site-packages/rucio/__init__.py'>
<module 'pyAMI' from '/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/pyAmi/pyAMI-5.1.3/lib/pyAMI/__init__.py'>
<module 'XRootD' from '/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/xrootd/5.5.3-x86_64-centos7/lib/python3.9/site-packages/XRootD/__init__.py'>
<module 'pandaclient' from '/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/PandaClient/1.5.40/lib/python3.6/site-packages/pandaclient/__init__.py'>
<module 'uproot' from '/home/feickert/bootstrap-example/romain-example/lib/python3.9/site-packages/uproot/__init__.py'>
<module 'pandas' from '/home/feickert/bootstrap-example/romain-example/lib/python3.9/site-packages/pandas/__init__.py'>


## bootstrap.sh
#!/bin/bash

# Run in a subshell as to not pollute the original environment
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
. "${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh" -3  # setupATLAS

lsetup 'panda'
. cvmfs-venv romain-example
# Need this to avoid rucio breaking pip later
# The PYTHONPATH is polluted with Python 3.6, so get the last pip release
# that was compatible with Python 3.6 (https://pypi.org/project/pip/21.3.1/)
python -m pip install --upgrade 'pip<22.0'
python -m pip show pip

## example.sh
#!/bin/bash

# Install cvmfs-venv
mkdir -p ~/.local/bin
export PATH=~/.local/bin:"${PATH}"  # If ~/.local/bin not on PATH already
curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv
chmod +x ~/.local/bin/cvmfs-venv

# Bootstrap a working environment first time
if [ ! -d "romain-example" ]; then
    if [ -f "bootstrap.sh" ]; then
        # Run in a subshell as to not pollute the current environment
        bash bootstrap.sh
    fi
fi

# Setup the real environment now
setupATLAS
lsetup 'rucio' 'panda' 'pyami' 'xrootd'
. romain-example/bin/activate
# Needed to avoid bug with rucio and polluted PYTHONPATH
# The PYTHONPATH is polluted with Python 3.6, so get the last pandas release
# that was compatible with Python 3.6 (https://pypi.org/project/pandas/1.1.5/)
python -m pip --quiet install --upgrade uproot 'pandas<1.2.0'

python --version --version
echo "PYTHONPATH=${PYTHONPATH}"
echo "# python -m pip list"
python -m pip list
echo "# python -m pip list --local"
python -m pip list --local

echo "command -v rucio: $(command -v rucio)"
echo "command -v xrootd: $(command -v xrootd)"
echo "command -v prun: $(command -v prun)"
echo "command -v ami: $(command -v ami)"
python import_example.py

## import_example.py
import rucio
import pyAMI
import XRootD
import pandaclient
import uproot
import pandas

print(rucio)
print(pyAMI)
print(XRootD)
print(pandaclient)
print(uproot)
print(pandas)
	#!/bin/bash

	# Run in a subshell as to not pollute the original environment
	export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
	. "${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh" -3 # setupATLAS

	lsetup 'panda'
	. cvmfs-venv romain-example
	# Need this to avoid rucio breaking pip later
	# The PYTHONPATH is polluted with Python 3.6, so get the last pip release
	# that was compatible with Python 3.6 (https://pypi.org/project/pip/21.3.1/)
	python -m pip install --upgrade 'pip<22.0'
	python -m pip show pip
	#!/bin/bash

	# Install cvmfs-venv
	mkdir -p ~/.local/bin
	export PATH=~/.local/bin:"${PATH}" # If ~/.local/bin not on PATH already
	curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv
	chmod +x ~/.local/bin/cvmfs-venv

	# Bootstrap a working environment first time
	if [ ! -d "romain-example" ]; then
	if [ -f "bootstrap.sh" ]; then
	# Run in a subshell as to not pollute the current environment
	bash bootstrap.sh
	fi
	fi

	# Setup the real environment now
	setupATLAS
	lsetup 'rucio' 'panda' 'pyami' 'xrootd'
	. romain-example/bin/activate
	# Needed to avoid bug with rucio and polluted PYTHONPATH
	# The PYTHONPATH is polluted with Python 3.6, so get the last pandas release
	# that was compatible with Python 3.6 (https://pypi.org/project/pandas/1.1.5/)
	python -m pip --quiet install --upgrade uproot 'pandas<1.2.0'

	python --version --version
	echo "PYTHONPATH=${PYTHONPATH}"
	echo "# python -m pip list"
	python -m pip list
	echo "# python -m pip list --local"
	python -m pip list --local

	echo "command -v rucio: $(command -v rucio)"
	echo "command -v xrootd: $(command -v xrootd)"
	echo "command -v prun: $(command -v prun)"
	echo "command -v ami: $(command -v ami)"
	python import_example.py
	import rucio
	import pyAMI
	import XRootD
	import pandaclient
	import uproot
	import pandas

	print(rucio)
	print(pyAMI)
	print(XRootD)
	print(pandaclient)
	print(uproot)
	print(pandas)