Skip to content

Instantly share code, notes, and snippets.

@tuxdna

tuxdna/README.md

Last active October 10, 2018 07:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tuxdna/767d39ef8f5c6aefee2ac238f8eeb6dc to your computer and use it in GitHub Desktop.
Save tuxdna/767d39ef8f5c6aefee2ac238f8eeb6dc to your computer and use it in GitHub Desktop.
Download ZIP
Mointoring NVIDIA GPUs
Raw
README.md

Mointoring NVIDIA GPUs

List NVIDIA GPUs available

$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader
GeForce GTX 1050

Monitor GPU stats ( temperature, utlilization, memory )

$ nvidia-smi --query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv -l 5

timestamp, name, pci.bus_id, driver_version, pstate, pcie.link.gen.max, pcie.link.gen.current, temperature.gpu, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2018/10/10 13:06:03.419, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:04.420, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:05.421, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:06.422, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:07.423, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:08.424, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:09.426, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:10.428, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:11.429, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:12.430, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4038 MiB, 2 MiB
2018/10/10 13:06:13.431, GeForce GTX 1050, 00000000:01:00.0, 396.45, P8, 3, 1, 40, 0 %, 0 %, 4040 MiB, 4028 MiB, 12 MiB
2018/10/10 13:06:14.432, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 41, 1 %, 0 %, 4040 MiB, 3975 MiB, 65 MiB
2018/10/10 13:06:15.433, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 0 %, 0 %, 4040 MiB, 3949 MiB, 91 MiB
2018/10/10 13:06:16.434, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 0 %, 0 %, 4040 MiB, 3923 MiB, 117 MiB
2018/10/10 13:06:17.434, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 2 %, 0 %, 4040 MiB, 3901 MiB, 139 MiB
2018/10/10 13:06:18.435, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 2 %, 0 %, 4040 MiB, 3875 MiB, 165 MiB
2018/10/10 13:06:19.435, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 0 %, 0 %, 4040 MiB, 3857 MiB, 183 MiB
2018/10/10 13:06:20.437, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 1 %, 0 %, 4040 MiB, 3835 MiB, 205 MiB
2018/10/10 13:06:21.438, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 42, 1 %, 0 %, 4040 MiB, 3825 MiB, 215 MiB
2018/10/10 13:06:22.439, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 43, 0 %, 0 %, 4040 MiB, 3817 MiB, 223 MiB
2018/10/10 13:06:23.439, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 43, 0 %, 0 %, 4040 MiB, 3817 MiB, 223 MiB
2018/10/10 13:06:24.441, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 43, 0 %, 0 %, 4040 MiB, 3817 MiB, 223 MiB
2018/10/10 13:06:25.441, GeForce GTX 1050, 00000000:01:00.0, 396.45, P0, 3, 3, 43, 0 %, 0 %, 4040 MiB, 3817 MiB, 223 MiB

GPU Current Clock setup

$ nvidia-smi -q -d CLOCK
==============NVSMI LOG==============

Timestamp                           : Wed Oct 10 13:02:09 2018
Driver Version                      : 396.45

Attached GPUs                       : 1
GPU 00000000:01:00.0
    Clocks
        Graphics                    : 139 MHz
        SM                          : 139 MHz
        Memory                      : 405 MHz
        Video                       : 544 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 1911 MHz
        SM                          : 1911 MHz
        Memory                      : 3504 MHz
        Video                       : 1708 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    SM Clock Samples
        Duration                    : 21.64 sec
        Number of Samples           : 20
        Max                         : 1354 MHz
        Min                         : 139 MHz
        Avg                         : 978 MHz
    Memory Clock Samples
        Duration                    : 21.62 sec
        Number of Samples           : 20
        Max                         : 3504 MHz
        Min                         : 405 MHz
        Avg                         : 2494 MHz
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A

Stats for Power and clock

$ nvidia-smi --query-gpu=index,timestamp,power.draw,clocks.sm,clocks.mem,clocks.gr --format=csv -l 1
index, timestamp, power.draw [W], clocks.current.sm [MHz], clocks.current.memory [MHz], clocks.current.graphics [MHz]
0, 2018/10/10 13:04:02.638, [Not Supported], 139 MHz, 405 MHz, 139 MHz
0, 2018/10/10 13:04:03.640, [Not Supported], 139 MHz, 405 MHz, 139 MHz
0, 2018/10/10 13:04:04.642, [Not Supported], 139 MHz, 405 MHz, 139 MHz
0, 2018/10/10 13:04:05.644, [Not Supported], 139 MHz, 405 MHz, 139 MHz

GPUs elsewhere consume how much power?

$ nvidia-smi --query-gpu=index,timestamp,power.draw,clocks.sm,clocks.mem,clocks.gr --format=csv -l 5
index, timestamp, power.draw [W], clocks.current.sm [MHz], clocks.current.memory [MHz], clocks.current.graphics [MHz]
1, 2018/10/10 08:40:48.924, 15.52 W, 139 MHz, 405 MHz, 139 MHz
1, 2018/10/10 08:40:53.937, 15.33 W, 139 MHz, 405 MHz, 139 MHz

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment