My singularity definition files are here ~/singularity/containers/ . There are two files : base.def which builds my base image (including all of my development tools), and a Chapel-specific image. I do most of my package installs with Spack.
These are somewhat rough notes, but could be further cleaned up for a more automatic build.
These images are also built to be run on an Infiniband system, so I’ve included the ibverbs package.
You’ll need to have the modules.yaml and runscript.sh files in the same directory as the
base.def file. This also adds in the tarballs for Intel MPI and MKL, plus a few files to build these.
For completeness, the files in the external-files/intel-mpi-mkl-v0/ directory are :
- l_mkl_2020.4.304.tgz
- l_mpi_2019.10.317.tgz
- packages.yaml
- silent.cfg
packages.yaml is to tell Spack about this, the file is
packages:
intel-mpi:
externals:
- spec: "intel-mpi@2019.10.317 arch=linux-ubuntu20.04-skylake"
prefix: /opt/intel
buildable: False
intel-mkl:
externals:
- spec: "intel-mkl@2020.4.304 arch=linux-ubuntu20.04-skylake threads=openmp"
prefix: /opt/intel
buildable: Falsesilent.cfg is to install this without user intervention. That file is
ACCEPT_EULA=accept ARCH_SELECTED=INTEL64 COMPONENTS=ALL CONTINUE_WITH_INSTALLDIR_OVERWRITE=yes CONTINUE_WITH_OPTIONAL_ERROR=yes PSET_INSTALL_DIR=/opt/intel PSET_MODE=install SIGNING_ENABLED=no
I chose to do these with an Ubuntu 20.04 base, for no other reason than that I am familiar with it.
To get MPI support, I needed to pick an MPI library for the container itself. I ended up using Intel-MPI since it looks like it had PMI support built in (necessary for easy interaction with SLURM). Using OpenMPI or MPICH would have required my to figure out how to include the necessary libraries with all of this.
The base image is useful to validate whether MPI is working within the container with SLURM etc.
Installed packages include:
- Intel MPI
- Intel MKL
- FFTW
- GSL
- HDF5
The Chapel image starts from the base image (so set your directory paths appropriately). I build with COMM=local, as well as COMM=gasnet-mpi and COMM=gasnet-ibv (the former for my laptop, the latter for our cluster).
Nothing special : sudo singularity build <name>.sif <name>.def
You’ll notice that I have a runscript set up in the initial code; this lets me set up my environment the way I want to (especially spack, which I couldn’t figure out how to get singularity to set up).
The runscript allows me to both go into a shell, or to directly run a command in the container. I tend to develop in a shell, and then run codes directly (see below)
On my laptop, I run with
singularity run --cleanenv --no-home --bind <tmpdir>:/home/$USER,$PWD:/data --pwd '/data' <commands>
- –cleanenv : doesn’t pass the environment. This is useful to avoid eg. modules in the container from picking up things from outside.
- –no-home : again, nice to keep this clean
- binds : $HOME inside the container is mapped to a temporary directory to avoid polluting a run directory with eg. .bash_history files. The current directory is mounted under /data, which is where the container starts (–pwd)
For mpi jobs, I do mpirun -np <num> singularity run ….
Running on the cluster requires a few modifications
- cleanenv doesn’t seem to work because it looks like eg. SLURM passes some information using environment variables. To avoid modules infecting the container, I have the container override certain env variables by setting appropriate SINGULARITYENV_<> variables.
- Intel MPI needs to find the host PMI library. I bind its location to /host/lib64 and set the I_MPI_PMI_LIBRARY to its value (for Intel MPI, it looks like you need the PMI-2 variant)
- bind /sys to /sys. On our cluster, we often end up assigning partial nodes. Chapel uses hwloc to figure out details of the system, and it appears that hwloc uses /sys.
- I sent the home directory to a local scratch directory visible to all nodes, instead of a local tmp directory.
I run an MPI job on the cluster with
srun -n <nloc> singularity run ...
- Using gasnet+mpi conduit; required GASNET_MPI_THREAD=multiple
- gasnet+mpi on Intel MPI with the container seems to require setting I_MPI_OFI_PROVIDER=sockets
- Intel MPI on my laptop seems to do a certain amount of CPU pinning to avoid some oversubscription.
- Need to bind /sys from the host to the container.
- Mount the location of the PMI library and set
I_MPI_PMI_LIBRARY I_MPI_DEBUG=4is a useful debug state to see what is happening.- I don’t seem to need to set
I_MPI_OFI_PROVIDER - I need to set
MLX5_SINGLE_THREADED=0to allow Chapel to run.