Use a CVMFS LCG view hybrid Python virtual environment for HTCondor submissions

README.md

Use a CVMFS LCG view hybrid Python virtual environment for HTCondor submissions

Create virtual environment

Generate a Python virtual environment using the Python runtime provided by a LCG view, and then from a high level requirements.txt file use pip-tools to compile an environment given by the accompanying lock file.

$ bash create_cvmfs_venv.sh

Write the HTCondor executable to work with the environment

Write the example.sh so that it sets up the same Python from the LCG view and then loads the same virtual environment. A key point is to make sure that the output equivalent of

export PYTHONPATH="$(readlink -f $(find venv -type d -iname site-packages)):${PYTHONPATH}"

is used. Also remember to make example.sh executable with

$ chmod +x example.sh

so that HTCondor can treat it as a regular executable.

Submit the HTCondor job

$ mkdir -p output error log  # Ensure output directories for job
$ condor_submit example.sub
$ condor_q

Job management

If you're trying to debug a job or need a reminder on how to get more information on jobs with condor_q check out the "Managing a Job" section of the HTCondor docs.

create_cvmfs_venv.sh

#!/bin/bash

# load LCG view to get Python
. /cvmfs/sft.cern.ch/lcg/views/LCG_101/x86_64-centos7-gcc11-opt/setup.sh
# Create Python virtual environment
python -m venv venv
. venv/bin/activate
# Ensure that the Python virtual environment is at head of PYTHONPATH
# c.f. https://github.com/matthewfeickert/cvmfs-venv for more info and alternatives to deal with this
export PYTHONPATH="$(readlink -f $(find venv -type d -iname site-packages)):${PYTHONPATH}"

python -m pip install --upgrade --ignore-installed pip setuptools wheel
python -m pip install --upgrade --ignore-installed pip-tools

# Generate envrionment lock file
pip-compile \
    --generate-hashes \
    --output-file requirements.lock \
    requirements.txt

# Use Brett Cannon's recommendations for pip-secure-install to ensure environment
# is reproducible and installed as securely as possible.
# c.f. https://www.python.org/dev/peps/pep-0665/#secure-by-design
# c.f. https://github.com/brettcannon/pip-secure-install
# c.f. https://twitter.com/brettsky/status/1486137764315688961
# As working in CVMFS LCG views also add --ignore-installed to avoid CVMFS packages
python -m pip install \
    --ignore-installed \
    --no-deps \
    --require-hashes \
    --only-binary :all: \
    --requirement requirements.lock

python -m pip list --local

example.py

import pyhf

model = pyhf.simplemodels.uncorrelated_background(
    signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
)
data = [51, 48] + model.config.auxdata
test_mu = 1.0
CLs_obs, CLs_exp = pyhf.infer.hypotest(
    test_mu, data, model, test_stat="qtilde", return_expected=True
)

print(f"Observed: {CLs_obs}, Expected: {CLs_exp}")

example.sh

#!/bin/bash

# Load environment
. /cvmfs/sft.cern.ch/lcg/views/LCG_101/x86_64-centos7-gcc11-opt/setup.sh  # load LCG view to get Python
. /afs/cern.ch/user/f/feickert/htcondor_example/venv/bin/activate  # load Python virtual environment
# Ensure that the Python virtual environment is at head of PYTHONPATH
# c.f. https://github.com/matthewfeickert/cvmfs-venv for more info and alternatives to deal with this
# This should be the equivalent of: export PYTHONPATH="$(readlink -f $(find venv -type d -iname site-packages)):${PYTHONPATH}"
export PYTHONPATH="/afs/cern.ch/user/f/feickert/htcondor_example/venv/lib/python3.9/site-packages:${PYTHONPATH}"

# Run script
python example.py

example.sub

executable            = example.sh
arguments             = $(ClusterId) $(ProcId)
output                = output/example.$(ClusterId).$(ProcId).out
error                 = error/example.$(ClusterId).$(ProcId).err
log                   = log/example.$(ClusterId).log

should_transfer_files = YES
transfer_input_files  = example.py

queue

requirements.txt

pyhf==0.6.3

submit.sh

#!/bin/bash

mkdir -p output error log
condor_submit example.sub
condor_q