Skip to content

Instantly share code, notes, and snippets.

@BramVanroy

BramVanroy/gpu-error-log.sh

Last active July 18, 2023 07:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save BramVanroy/d90f66e19b6f734ba43dc1cbb3af2b64 to your computer and use it in GitHub Desktop.
Save BramVanroy/d90f66e19b6f734ba43dc1cbb3af2b64 to your computer and use it in GitHub Desktop.
Download ZIP
Log ssh GPU errors
Raw
gpu-error-log.sh
# If there is an error in nvidia-smi, log it to a file in ~/gpu-errors!
nvidia_smi_output=$(nvidia-smi)
if echo "nvidia_smi_output" | grep -q "ERR"; then
fname=~/gpu-errors/$(hostname)-error.txt
pdir=$(dirname "$fname")
mkdir -p "$pdir"
nvcc_output=$(nvcc --version)
echo "$nvidia_smi_output"$'\n'"$nvcc_version_output" > "$fname"
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment