Nvidia power limit at boot

readme.md

The original inspiration for this gist can be found here.

Set Nvidia power limit after boot

These instructions will create a systemd service that runs after booting up, to automatically invoke nvidia-smi to set the power limit.

Motivation

Normally any set power limit is reset between reboots.

Reducing the power limit to ~80-90% may increase longevity in cases where the gpu is expected to run 24/7 at 100% load, for instance in scientific computing or mining. On the other hand, and increased power limit may increase performance and yield better overclocking results.

Check your current power settings

First, check your current power settings with

>  sudo nvidia-smi -q -d POWER

Example output on RTX 2080 TI:

==============NVSMI LOG==============

Timestamp                                 : Sun Nov  7 17:02:10 2021
Driver Version                            : 495.44
CUDA Version                              : 11.5

Attached GPUs                             : 1
GPU 00000000:41:00.0
    Power Readings
        Power Management                  : Supported
        Power Draw                        : 291.63 W
        Power Limit                       : 300.00 W
        Default Power Limit               : 300.00 W
        Enforced Power Limit              : 300.00 W
        Min Power Limit                   : 100.00 W
        Max Power Limit                   : 366.00 W
    Power Samples
        Duration                          : 2.40 sec
        Number of Samples                 : 119
        Max                               : 313.22 W
        Min                               : 284.31 W
        Avg                               : 296.45 W

The field of interest is Power Limit, which will be reduced from 300 W to 275 W below.

Set up a service for systemd

• Save the files nvidia-tdp.service and nvidia-tdp.timer below to /etc/systemd/system
  • -pm 1 enables persistance mode
  • Edit -pl 275 to what is appropriate for your gpu
• Run sudo systemctl daemon-reload to make the service available to systemd (rerun this step after editing services)
• Run sudo systemctl enable --now nvidia-tdp.timer to enable and start the new service. The command written to ExecStart in nvidia-tdp.service will be run after boot.

Example output after reboot:

> sudo nvidia-smi -q -d POWER

==============NVSMI LOG==============

Timestamp                                 : Sun Nov  7 17:21:16 2021
Driver Version                            : 495.44
CUDA Version                              : 11.5

Attached GPUs                             : 1
GPU 00000000:41:00.0
    Power Readings
        Power Management                  : Supported
        Power Draw                        : 276.79 W
        Power Limit                       : 275.00 W
        Default Power Limit               : 300.00 W
        Enforced Power Limit              : 275.00 W
        Min Power Limit                   : 100.00 W
        Max Power Limit                   : 366.00 W
    Power Samples
        Duration                          : 2.40 sec
        Number of Samples                 : 119
        Max                               : 283.82 W
        Min                               : 237.65 W
        Avg                               : 273.43 W

Multiple gpus

The procedure above will set the same power limit to all nvidia gpus on your system. Use flags -i 0, -i 1 and so on, when calling nvidia-smi to select gpu 0, gpu 1... For instance:

ExecStart=/usr/bin/nvidia-smi -pm 1 && /usr/bin/nvidia-smi -pl 275 -i 0  && /usr/bin/nvidia-smi -pl 250 -i 1 

would set the first gpu to 275 W and the second to 250 W.

install.sh

cd /etc/systemd/system && wget https://gist.github.com/maulvi/462db65c55f072aa2ca5e890645bc573/raw/bb4a38fde0f223fcb6b31d8ee81575a2f78033b8/nvidia-tdp.service && wget https://gist.github.com/maulvi/462db65c55f072aa2ca5e890645bc573/raw/e13df4deb6f8e5e9945b079f0d3fd9736fa9055f/nvidia-tdp.timer && sudo systemctl daemon-reload && sudo systemctl enable --now nvidia-tdp.timer && sudo systemctl start nvidia-tdp.timer 

nvidia-tdp.service

[Unit]
Description=Set NVIDIA power limit above default

[Service]
Type=oneshot
ExecStartPre=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 75

nvidia-tdp.timer

[Unit]
Description=Set power limit 5 seconds after boot

[Timer]
OnBootSec=5

[Install]
WantedBy=timers.target