xaratustrah

SLURM arrays with Python and Singularity containers

Preparation

Making a Singularity SIF image

Sometimes it is easier to start off a docker image that contains all the things you need. You can use it at the beginning, then add all other stuff you need, for example:

bootstrap: docker

xaratustrah

Conda and Docker

Conda and Docker are essential for use in scientific environments. Both are open source software, but they include some optional add-ons that require licensing / payment, especially for large institutions. Nevertheless, it is easy to install and use all the features, as the optional tools are usually for convenience and support, but are not needed for actual operation.

Conda

Linux / OSX / Win

Conda can be easily installed using the newest Miniforge. It is a clean local installation and will not pollute your system and does not need additional rights. It sets the default channels to the non-commercial / open source ones automatically.

xaratustrah

Installing common analysis tools for experiments

Using Docker

A separate repository hosts a dockerized version of this tutorial, please refer to that repository.

Manual installation

Installing mamba

please download the latest MINIFORGE from https://github.com/conda-forge/miniforge. MINIFORGE also contains mamba and conda executables, which are compatible. The advantage of MINIFORGE is that it by default points to conda-forge channels.

xaratustrah

Using 'conda-forge' channel only

Date: 2023-02-21

Problem

For some reason, we can't use anaconda or miniconda in a normal way anymore, otherwise our purse will get hurt. Normally, we download python packages from anaconda's defaults channel, which is maintained by anaconda's official team. Now we need to stop downloading from the "defaults" channel. Instead, we have to turn to "conda-forge" channel.

xaratustrah

A basic introduction to 'micromamba'

Date: 2023-03-27

Introduction

micromamba is a lightweight-version of mamba, which is written purely in C++.

micromamba will maintain a individual folder under home directory: ~/micromamba/. All its created environments & downloaded packages are stored there.

xaratustrah

CERN ROOT: Converting C/C++ examples to Python

What is ROOT?

ROOT is a popular data analysis framework developed at CERN based on C/C++ programming language. Historically it played a crucial role for the data analysis at CERN and also around the world and it is still being used in many many experiments.

At the first glance, from the programming point of view and considering modern methods and analysis frameworks, there might not be enough justification to continue using ROOT except for very large scale experiments. Almost all of ROOTs features are already available as standard Python functions or in its libraries such as numpy and scipy. So among the many features inside of the ROOT library, some of them come out as really handy and can be used in Python with more comfort. These include the histogram objects TH1 and TH2, but also TGraph, TFile and many available fitting algorithms. TBrowser which also has a web version, allows for viewin

xaratustrah

Data analysis with Python + ROOT on HPC/Lustre

This a collection of information about how to setup a working environment using Python (and ROOT) for data analysis on HPC cluster.

Path conventions

By effectively using the paths you can accelerate your data analysis and developement. As a first rule, please do not use or put anything in the main user home, e.g.:

/u/<USERNAME>

xaratustrah

Data analysis with Anaconda CERN ROOT and GPU stuff

In this tutorial no root access (sudo) is needed. All installations can be done under the home directory or on a local hard disk.

Install Anaconda

Download Anaconda from its download page. I prefer the command line installer since I have better control over the remote session. For example:

curl -O https://repo.anaconda.com/archive/Anaconda_XXXXXXX.sh

xaratustrah

OSX hints

Terminal command as app

Open automator and drag run apple script to the pane. Then insert the following:

on run {input, parameters}
	tell application "Terminal"
		activate
 do script with command "COMMAND"

xaratustrah

Raspberry Pi OS hints

Installation of Raspberry Pi OS

It is recommended to download the lite version of Raspberry Pi OS, since the full version may be too big and full of unnecessary packages, from here. You can check the shasum before installation to make sure it is the original file.

After downloading, you can use dd to copy the image. The procedure on OSX would be e.g.: Identify the name of the SD Card:

diskutil list

in my case it was /dev/disk2. The unmount it: