HPCToolkit Sanitizer spack install

sanitizer.md

Install spack

git clone https://github.com/spack/spack.git
export SPACK_ROOT=/path/to/spack
export PATH=${SPACK_ROOT}/bin:${PATH}
source ${SPACK_ROOT}/share/spack/setup-env.sh

Clone HPCToolkit

git clone git@github.com:Jokeren/hpctoolkit.git
cd hpctoolkit
git checkout sanitizer

Configure spack

• config.yaml

cd spack/etc/spack
cp defaults/config.yaml .
vi config.yaml [change build_job to number of cores on your machine]

• packages.yaml

cd spack/etc/spack 
cp /path/to/hpctoolkit/spack/packages.yaml ./
vi config.yaml [change packages settings, including cuda@10.1, dyninst@10.1, cmake, perl, gcc@7.3.0]

• package.py

cd var/spack/repos/builtin/packages/hpctoolkit 
cp /path/to/hpctoolkit/spack/package.py ./

• check

spack spec hpctoolkit

Install hpctoolkit

spack install --only dependencies hpctoolkit 
cd /path/to/hpctoolkit
mkdir build && cd build
../configure --prefix=/path/to/install/prefix --with-cuda=/usr/local/cuda-10.1 --with-cupti=/usr/local/cuda-10.1/extras/CUPTI --with-cuda-sanitizer=/usr/local/cuda-10.1/extras/Sanitizer --with-spack=/path/to/spack/libraries/root --enable-develop
[Tip: check spack libraries' root->spack find --path package]
make install -j8

Install memory patch

git clone git@github.com:Jokeren/hpctoolkit-gpu-memory-patch.git
cd hpctoolkit-gpu-memory-patch
make

Test sanitizer

git clone git@github.com:Jokeren/hpctoolkit-gpu-samples.git
cd hpctoolkit-gpu-samples/cuda_vec_add
export OMP_NUM_THREADS=1 [you can set any number of threads as you want]
cp /path/to/hpctoolkit-gpu-memory-patch/memory.fatbin ./
hpcrun -e nvidia-cuda-memory ./main &> log [you can enable block sampling by nvidia-cuda-memory@sampling frequency]
more log