Integrate Spack modules in conda environment.

conda_spack.md

About

Automatic loading of Spack modules when the Conda environment is loaded.

Helper function

Spack and Conda can influence each other with side effects. Therefore I do not activate them by default. The following functions allow you to activate Conda and Spack by bash command.

# for example, add to .bashrc

enabled_spack=0
enable_spack(){
    if [[ enabled_spack -eq 0 ]]; then
	. /opt/spack/share/spack/setup-env.sh
	enabled_spack=1
    fi
}

enabled_conda=0
enable_conda(){
    if [[ enabled_conda -eq 0 ]]; then
	# >>> conda initialize >>>
	# !! Contents within this block are managed by 'conda init' !!
	__conda_setup="$('/opt/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
	if [ $? -eq 0 ]; then
	    eval "$__conda_setup"
	else
	    if [ -f "/opt/miniconda3/etc/profile.d/conda.sh" ]; then
		. "/opt/miniconda3/etc/profile.d/conda.sh"
	    else
		export PATH="/opt/sehrig/miniconda3/bin:$PATH"
	    fi
	fi
	unset __conda_setup
	# <<< conda initialize <<<
	enable_conda=1
    fi
}

# wrapper command to automatically activate Spack when a specific Spack module is required
alias singularity=my_singularity
loaded_spack=0
my_singularity(){
    if [[ loaded_spack -eq 0 ]]; then
	if [[ enabled_spack -eq 0 ]]; then
	    enable_spack
	fi
	spack load singularity@3.5.3
	loaded_spack=1
    fi
    command singularity $@
}

Create environment and set activate and deactivate script

Create conda environment

conda create -n test-env

Create activate and deactivate scripts

mkdir -p /opt/miniconda3/envs/test-env/etc/conda/activate.d
mkdir -p /opt/miniconda3/envs/test-env/etc/conda/deactivate.d

nano /opt/miniconda3/envs/test-env/etc/conda/activate.d/spack_load.sh
nano /opt/miniconda3/envs/test-env/etc/conda/activate.d/spack_unload.sh

spack_load.sh

#!/bin/bash

if [[ enabled_spack -eq 0 ]]; then
    enable_spack
fi

spack load cuda@8.0.61
spack load llvm@9.0.0

spack_unload.sh

#!/bin/bash

spack unload llvm@9.0.0
spack unload cuda@8.0.61

Since this PR, spack does not set the LD_LIBRARY_PATH automatically anymore. This solves and causes problems. A valid solution is, that spack generates a SPACK_LD_LIBRARY_PATH variable and the conda init script copies the required paths to the LD_LIBRARY_PATH variable.

Following commands needs to run one time, that spack creates the SPACK_LD_LIBRARY_PATH variable:

spack config add modules:prefix_inspections:lib64:[SPACK_LD_LIBRARY_PATH]
spack config add modules:prefix_inspections:lib:[SPACK_LD_LIBRARY_PATH]

The following conda init script set the path to the CUDA lib64 folder:

export BEFORE_CONDA_ACTIVATE_LD_LIBRARY=$LD_LIBRARY_PATH

directories=$(echo $SPACK_LD_LIBRARY_PATH | tr ":" "\n")
for directory in $directories
do
    if [[ $directory =~ "cuda" ]]; then
    LD_LIBRARY_PATH=$directory:$LD_LIBRARY_PATH
    fi
done

export LD_LIBRARY_PATH

deinit script:

export LD_LIBRARY_PATH=$BEFORE_CONDA_ACTIVATE_LD_LIBRARY
unset BEFORE_CONDA_ACTIVATE_LD_LIBRARY