Skip to content

Instantly share code, notes, and snippets.

@johnnychen94
Last active October 3, 2018 18:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save johnnychen94/eae88c335881436bc4a43bdfaeb5370e to your computer and use it in GitHub Desktop.
Save johnnychen94/eae88c335881436bc4a43bdfaeb5370e to your computer and use it in GitHub Desktop.
[kill tensorflow] killall ipykernel processes that occupies GPU memory for a long time
#! /bin/bash
MAX_COUNT=60
COUNT=0
while [ 1 ]
do
FREE=`nvidia-smi -q | grep Free -m 1 | grep -o '[0-9]*'`
GPU=`nvidia-smi -q | grep Gpu -m 1 | grep -o '[0-9]*'`
echo $COUNT/$MAX_COUNT
if [ $FREE -lt 1000 ] && [ $GPU -lt 10 ] ; then
let "COUNT = COUNT + 1"
if [ $COUNT -gt $MAX_COUNT ]; then
echo "Killall ipykernel to free the gpu memory"
ps -ef | grep ipykernel | grep -v grep | cut -c 9-15 | xargs kill -9
fi
else
let "COUNT = 0"
fi
sleep 10
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment