This script enhances the functionality of nvidia-smi and provides the following information:
Usernamefull process CommandGPU IDPID
This is useful on multi-user servers and can be used to quickly identify which user is using the GPU and running what kind of program.
nvidia-smi && (nvidia-smi |tr -s ' '|grep -Eo "| [0123456789]+ N/A N/A [0-9]{3,} .*"|awk -F' ' '{system("s=$(cat /proc/"$4"/cmdline| tr \"\\0\" \" \");u=$(ps -o uname= -p "$4");echo "$1"sep"$4"sep$u sep"$7"sep$s" ) }'|sed 's/sep/\t/g')The output is like this
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3296 C python 5825MiB |
| 0 21266 C /opt/conda/bin/python 6319MiB |
| 1 3296 C python 137MiB |
| 1 8063 C /usr/bin/python 273MiB |
| 1 21266 C /opt/conda/bin/python 4735MiB |
| 1 30439 C /usr/bin/python 199MiB |
| 2 3296 C python 137MiB |
| 2 21266 C /opt/conda/bin/python 4735MiB |
+-----------------------------------------------------------------------------+
0 3296 user1 5825MiB python playground2.py
0 21266 user2 6319MiB /opt/conda/bin/python -u /cail/model.py
1 3299 root 137MiB python playground2.py
1 8063 root 273MiB /usr/bin/python -u /sentsim/cli_predict.py
1 21266 root 4735MiB /opt/conda/bin/python -u /cail/model.py
1 30439 root 199MiB /usr/bin/python -u /sentsim/train.py
2 3296 root 137MiB python playground2.py
2 21266 root 4735MiB /opt/conda/bin/python -u /cail/model.py
For convenience, you can alias the command.