Setting up a developers area for new python package structure in the GEM online software environment

dev-setup.md

Instructions to set up a working developers area

• Clone the repository you'll be doing developments on
  • For most repositories other than cmsgemos, you should do the following to bring in the submodules
    • git submodule init
    • git submodule update
  • Development will, in general, not be supported on non GEM DAQ machines, so if you choose to do so, support will only be provided on a best effort basis
• If you will be doing developments that depend on external packages, find the appropriate version to test against
  • gemplotting
  • vfatqc
  • cmsgemos
• Take the appropriate tar.gz file for the package
  • gempython_vfatqc-2.0.0.tar.gz
  • gempython_gemplotting-1.0.0.tar.gz
  • cmsgemos
• Set up a python virtual environment somewhere:

python -m virtualenv -p <python version> --system-site-packages /path/to/your/venv
. /path/to/your/venv/bin/activate
python -m pip install -r requirements-dev.txt
# equivalent to the following
# python -m pip install -U importlib setuptools pip
# with a python2.6 virtualenv on slc6 default, uncomment the following line
# python -m pip install -I importlib 'setuptools<39.2.0'  'pip<10.1.0'
# install the necessary, external, unmodified dependencies
python -m pip install <gempython package>.tar.gz --no-deps

# set up the environment
export PATH=$VIRTUAL_ENV/lib/python*/site-packages/gempython/scripts:$PATH
export PATH=$VIRTUAL_ENV/lib/python*/site-packages/gempython/gemplotting/macros:$PATH

# create the mapping files
find <path/to/venv/lib/../site-packages/gempython> -type f -name buildMapFiles.py -exec python {} \;

• You can do your development inside a virtualenv with no problem
  • Due to the tree structure of the virtualenv, trying to make rpm a package from inside a virtualenv will not currently work
  • A solution for this will be investigated, but recommend to not try to build from inside the virtualenv, rather use it for testing the built package
  • A currently untested workaround is a new make pip target, which will only build the bdist/sdist tarballs
• When you are ready to test your changes to the package you are working on
  • Compile the changes and create a new package

make && make pip

• Install the new package into your virtualenv

pip install -I rpm/<package_updated>.tar.gz

@bdorney Your script doesn't work with bash:

$ source setup_gemdaq.sh -p dorney_script_test_env
VENV_ROOT
you must provide the -venv argument
Usage: -bash [options] -p </path/to/venv/location> Options: -c cmsgemos release version (e.g. X.Y.Z) -g gemplotting release version (e.g. X.Y.Z) -G gemplotting dev version (e.g. single integer) -h displays this string -v vfatqc release version (e.g. X.Y.Z) -V vfatqc dev version (e.g. single integer) -w No value following, deletes and recreates the venv from scratch The virtualenv found at -p will either be activated or created

Same in zsh:

% source setup_gemdaq.sh -p dorney_script_venv
VENV_ROOT dorney_script_venv
ELOG_PATH not set, please set ELOG_PATH to a directory where plots created by analysis applications will be written
 (export ELOG_PATH=<your>/<elog>/<directory>/) and then rerun this script

Following your script by hand on gem-dqm01:

• Line 103: python -m pip install -U importlib setuptools pip

  • Should be pip install -I importlib 'setuptools<39.2.0' 'pip<10.1.0' for Python 2.6
  • Network isn't accessible, so packages have to be downloaded by hand
  • To download the packages, use pip download importlib 'setuptools<39.2.0' 'pip<10.1.0'
  • Then scp the files to cmsusr
  • The installation becomes pip install -I importlib-1.0.4.zip setuptools-36.8.0-py2.py3-none-any.whl pip-9.0.3-py2.py3-none-any.whl

• LL 111, 124, 140: cp /afs/cern.ch/... and wget: /afs and network aren't accessible

• Line 113: python -m pip install cmsgemos_gempython-0.3.1.tar.gz --no-deps -> pip install cmsgemos_gempython-${CMSGEMOS_VERSION}.tar.gz --no-deps

• I can't install anything in the venv:

  (dorney_venv)[lmoureau@srv-s2g18-34-01 ~]$ pip install --no-deps cmsgemos_gempython-0.3.1.tar.gz 
  -bash: /cmsnfshome0/nfshome0/lmoureau/dorney_venv/bin/pip: No such file or directory
  (dorney_venv)[lmoureau@srv-s2g18-34-01 ~]$ python -m pip install --no-deps cmsgemos_gempython-0.3.1.tar.gz 
  /cmsnfshome0/nfshome0/lmoureau/dorney_venv/bin/python: pip is a package and cannot be directly executed
  

  (this had already been reported here)

  • Worked around by creating <venv>/bin/pip:

    #!/usr/bin/env python
    
    # -*- coding: utf-8 -*-
    import re
    import sys
    
    from pip import main
    
    if __name__ == '__main__':
        sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
        sys.exit(main())
    

    (this is a copy of /usr/bin/pip with the shebang changed to use /usr/bin/env)
  • Installed cmsgemos and gemplotting* packages without problem

Same in zsh:
% source setup_gemdaq.sh -p dorney_script_venv
VENV_ROOT dorney_script_venv
ELOG_PATH not set, please set ELOG_PATH to a directory where plots created by analysis applications will be written
(export ELOG_PATH=///) and then rerun this script

@lmoureaux This is working as intended, you need to set $ELOG_PATH as it is stating. For the bash issue I suspect it was a lack of OPTIND being correctly set so that getopts would work correctly when sourced; please try again.

...for Python 2.6

Whether 2.6 is supported depends on when the p5 machines will be migrated to cc7. If the time scale is long then okay we can think of adding support; but otherwise I think this is a moot point. You can use the cc7 virtual machine at P5 which defaults to python2.7.

Network isn't accessible, so packages have to be downloaded by hand

That's not an issue with the script. You need to either setup network configuration to have port forwarding https or download the packages by hand (as is done when using setup_ctp7.sh).