Nvidia GPU Power Limits for Linux System

limit_gpu_power.md

You can use the following commands to limit the maximum power allowed to your NVIDIA GPU when using linux systems.

First, let's check how much power the GPU is allowed to draw and the current value:

nvidia-smi -q -d POWER

This should return an output similar to this one:

==============NVSMI LOG==============

Timestamp                                 : Fri Jul 12 17:27:17 2024
Driver Version                            : 555.42.06
CUDA Version                              : 12.5

Attached GPUs                             : 1
GPU 00000000:09:00.0
    GPU Power Readings
        Power Draw                        : 73.57 W
        Current Power Limit               : 450.00 W
        Requested Power Limit             : 450.00 W
        Default Power Limit               : 450.00 W
        Min Power Limit                   : 150.00 W
        Max Power Limit                   : 480.00 W
    Power Samples
        Duration                          : 2.36 sec
        Number of Samples                 : 119
        Max                               : 75.79 W
        Min                               : 73.21 W
        Avg                               : 73.80 W
    GPU Memory Power Readings 
        Power Draw                        : N/A
    Module Power Readings
        Power Draw                        : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A

Then, to actually apply the desired limit you can use the command below:

sudo nvidia-smi -pl 370

In this example we are setting the power limit to 370w, which is more than enough for games and should yield enough power for most Machine Learning workloads without setting your house on fire.

OBS: Feel free to adjust this value based on your needs and don´t take my word as law, do some research before doing any changes to your GPU, I'm not responsible for any damages for your hardware.

You can check if the new limits have been applied by executing the POWER command again:

nvidia-smi -q -d POWER

==============NVSMI LOG==============

Timestamp                                 : Fri Jul 12 17:27:44 2024
Driver Version                            : 555.42.06
CUDA Version                              : 12.5

Attached GPUs                             : 1
GPU 00000000:09:00.0
    GPU Power Readings
        Power Draw                        : 32.89 W
        Current Power Limit               : 370.00 W
        Requested Power Limit             : 370.00 W
        Default Power Limit               : 450.00 W
        Min Power Limit                   : 150.00 W
        Max Power Limit                   : 480.00 W
    Power Samples
        Duration                          : 2.36 sec
        Number of Samples                 : 119
        Max                               : 36.05 W
        Min                               : 31.43 W
        Avg                               : 33.08 W
    GPU Memory Power Readings 
        Power Draw                        : N/A
    Module Power Readings
        Power Draw                        : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A

Persisting the Changes

The recommended way to persist these changes to the system is to enable the driver persistence using the following command:

sudo nvidia-smi -pm 1

Expect this output:

Enabled Legacy persistence mode for GPU 00000000:09:00.0.
All done.

A known issue with this approach is that for some reason the power configuration is lost when you restart the machine, meaning you will need to re-apply this configuration everytime you start your computer, to avoid this, let's create a custom service to execute automatically.

Create a new file called /etc/systemd/system/nvidia-power-limit.service and paste the following content:

[Unit]
Description=Set NVIDIA GPU Power Limit

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pl 370

[Install]
WantedBy=multi-user.target

WARNING: Update the value to match your needs

Save the file and enable the new service:

sudo systemctl enable nvidia-power-limit.service

References

These instructions are based on the following blog post from LinuxConfig and I just reviewed if they were still valid and translated to something easier to access and remember, but they are the ones that deserve the credit and praise:

• https://linuxconfig.org/how-to-set-nvidia-power-limit-on-ubuntu