matthewfeickert

Use VS Code to run notebooks on remote servers in environments built on top of LCG views

(Note that this whole thing is a bit cursed as LCG views are a terrible way to distribute Python environments)

Setup

Get a working VS Code server on the remote

Requires:

matthewfeickert

Use GMail's SMTP servers to use CERN email account as an alias

1. Enable [forwarding of emails from Outlook][outlook-forwarding] to GMail

   1. In Outlook select Settings > Mail > Forwarding.
   2. Select "Enable forwarding".
   3. Enter your GMail address in the box under "Forward my email to:".
   4. (Optional, but recommended) Select "Keep a copy of forwarded messages"

[![outlook-view][outlook-view-gist]][outlook-forwarding]

matthewfeickert

Unexpected crashing of VS Code when opening Python files

Today (2023-04-03) VS Code is crashing whenever a Python file is opened in the mixed language repository https://gitlab.cern.ch/gstark/pycolumnarprototype

This behavior has not been observed until today. The machine uptime has been minimal.

$ uptime
 17:57:25 up 14:11,  1 user,  load average: 2.04, 2.15, 2.04

matthewfeickert

Bootstrap example

Bootstrap example to get around bugs with panda and rucio on CVMFS and PYTHONPATH pollution as described in the March 2023 PATHelp email thread "[ROOT undefined symbol error when setting up ROOT and Rucio together after setupATLAS][PATHelp-thread]".

Example

The PYTHONPATH is polluted with Python 3.6, so make an environment with the last pip and pandas releases that were compatible with Python 3.6.

The boostrap.sh Bash script sets up a Python virtual environment that has a version of pip that won't break with Python 3.6 which is further used in example.sh.

matthewfeickert

Failure of not using a virtual environment in installPip

By not using a virtual environment installPip makes it unable to interact with the virtual environment manually as one can not reliably manually install or uninstall packages.

Example: Unable to uninstall numpy

$ setupATLAS
...
$ asetup AnalysisBase,22.2.110

matthewfeickert

20223-03-15 technical response to JIRA ticket ATLAS ASGATLASG-2477: python packages for atlasexternals

As a warning, this is going to be a long response as it is going to need to discuss the technical details of how the way LCG views and the current way ATLAS externals works is fundamentally at odds with the core concepts of Python virtual environments. To make this more digestible in a JIRA ticket thread and to avoid repeating myself (somewhat) I am going to assume that anyone who is reading this will read the README I've already written for cvmfs-venv which explains this at a high level. Also, as I find JIRA's lack of Markdown to make reading next to anything on it pretty terrible I've put the full contents of the technical parts that follow in this GitHub Gist in case you want to read it

matthewfeickert

Creating a Jupyter kernel for Python virtual environment on UChicago Analysis Facility

• Reproduction of core content of ATLAS Talk Topic: Preferred way of creating a Jupyter kernel for Python virtual environment (ATLAS Internal, sorry)
• Corresponding YouTube explainer video: Creating an IPython kernel for your venv on UChicago AF Jupyter Lab (no captions, sorry)

For robustness and reproduciblity, I would like to use a Python virtual environment when working with ml_platform:latest containers (easy enough) but I would also like to have IPython kernels that I make in that virtual environment be recognized by Jupyter Lab's launcher. I believe that the easiest way to do this is to just use the --user flag when creating an IPython kernel, but I wanted to know if there is an alternative / create this as an example Issue for others to reference.

Here's an exam

matthewfeickert

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python

matthewfeickert

Detailed preservation of data and analyses enables their re-use beyond the scope of the original analyses.
This includes reinterpreting the results in combination with the outputs of other analyses (e.g. in global fits), reusing one or several existing analyses for testing new theoretical ideas (recasting), or reusing experimental or simulation data for a completely new analysis.
Designing and implementing datasets and analyses with this reuse in mind helps guide the pragmatic choices for where preservation effort is best spent.

Effective reinterpretation and recasting requires the preservation of both analysis data products and analyses, though the goals of the reinterpretation may require different levels of fidelity of the preservation.
For example, ATLAS has implemented full fidelity analysis reinterpretations internally using the RECAST framework and fully preserved analysis workflows.
CMS has similarly implemented much more lightweight solutions, with a smaller scope focus of statistically combining an

matthewfeickert

How to properly escape regex for pytest filterwarnings?

Hi. This question is motivated by scikit-hep/pyhf#1873 and is similar to subject matter to pytest-dev/pytest#8759 but I think different enough that I'm still going to ask.

Here's a GitHub Gist with the example files as well.

In scikit-hep/pyhf#1873 I have a set of warnings that I want to be able to filter with filterwarnings and with a simple regex that works fine with warnings.filterwarnings

# std_filterwarnings.py