Skip to content

Instantly share code, notes, and snippets.

@vsoch
Created Apr 24, 2022
Embed
What would you like to do?
Example of CiteLang render (post render)

Summary

Portability and reproducibility of complex software stacks is essential for researchers to perform their work. High Performance Computing (HPC) environments add another level of complexity, where possibly conflicting dependencies must co-exist. Although container technologies like Singularity @conda{name=singularity} make it possible to "bring your own environment," without any form of central strategy to manage containers, researchers who seek reproducibility via using containers are tasked with managing their own container collection, often not taking care to ensure that a particular digest or version is used. The reproducibility of the work is at risk, as they cannot easily install and use containers, nor can they share their software with others.

Singularity Registry HPC (shpc) @pypi{name=singularity-hpc} is the first of its kind to provide an easy means for a researcher to add their research software for sharing and collaboration with other researchers to an existing collection of over 200 popular scientific libraries @github{name=autamus/registry} @github{name=spack/spack, release=0.17}. The software installs containers as environment modules that are easy to use and read documentation for, and exposes aliases for commands in the container that the researcher can add to their pipeline without thinking about complex interactions with a container. The simple addition of an entry to the registry maintained by shpc comes down to adding a yaml file, and after doing this, another researcher can easily install the same software, down to the digest, to reproduce the original work.

References

Manager Name Credit
conda singularity 0.12
pypi singularity-hpc 0.12
github autamus/registry 0.12
github spack/spack 0.12
pypi pytest 0.02
pypi requests 0.02
pypi ruamel.yaml 0.02
pypi jsonschema 0.02
pypi Jinja2 0.02
pypi spython 0.02
github actions/checkout 0.02
github actions/upload-artifact 0.02
github docker/build-push-action 0.02
github docker/login-action 0.02
github docker/setup-buildx-action 0.02
github docker/setup-qemu-action 0.02
conda libgcc-ng 0.02
github autamus/artifact-dl 0.01
github autamus/binoc 0.01
github autamus/buildconfig 0.01
github autamus/builder 0.01
github autamus/librarian 0.01
github autamus/merge-commits 0.01
github autamus/smuggler 0.01
github NextThought/sphinxcontrib-programoutput 0.01
github actions/download-artifact 0.01
github actions/setup-python 0.01
github codecov/codecov-action 0.01
github dorny/paths-filter 0.01
github readthedocs/sphinx_rtd_theme 0.01
github sphinx-doc/sphinx 0.01
github ztane/python-Levenshtein 0.01
conda openssl 0.01
conda libuuid 0.01
conda libstdcxx-ng 0.01
conda libseccomp 0.01
conda libarchive 0.01
conda cni-plugins 0.01

Note that credit values are rounded and expanded (so shared dependencies are represented as one record) and may not add to 1.0. Rounded values that hit zero are removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment