Francesco149/pitcairn_linux_clock_fix.md

## pitcairn_linux_clock_fix.md

      
    Raw
  

              pitcairn_linux_clock_fix.md
            
          
    Gain massive GPU performance for Southern Islands AMD GPUs on Linux by removing DPM quirks in the kernel

I recently found out that my r9 270x GPU was not hitting full clock speeds because it was being throttled by the kernel based on old bug reports on DPM stability. This was an easy fix on gentoo where recompiling the kernel is trivial, but on ubuntu and similar, rebuilding the kernel is not so straightforward, so here's a guide based on ubuntu 16.04 LTS. If you run gentoo or know what you're doing you can just skip to the relevant code.
On my r9 270x, this resulted in an overall improvement in frame-rates of up to 50%: Unigine Valley went from something like 30-40 fps average to 55-60 and osu! (running in wine) went from 0.33 ms (~3000fps) to 0.25ms (~4000fps).
Note that I'm no linux god and all this info was pieced together by googling and asking on forums (credits to the guys at phoronix for pointing me to the quirks code).
If you're using the proprietary amdgpu-pro driver, read until the end of the guide, there's a final section on tweaking that one.
Let's check our max gpu clocks by forcing the gpu to a high power state and reading from debugfs:
(replace "radeon" with "amdgpu if you're using amdgpu)
loli@ubongo:~$ sudo su
[sudo] password for loli: 
root@ubongo:/home/loli# echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level
root@ubongo:/home/loli# cat /sys/kernel/debug/dri/0/radeon_pm_info uvd    vclk: 0 dclk: 0
power level 3    sclk: 85000 mclk: 90000 vddc: 1300 vddci: 1025 pcie gen: 3
root@ubongo:/home/loli# echo auto > /sys/class/drm/card0/device/power_dpm_force_performance_level

This is way lower than the 1070MHz Core, 1400 MHz memory my GPU should be clocking at.
First of all, we need to enable source repositories so we can actually get the kernel source

"Software & Updates" -> tick "Source code"
Close
Accept the reload

Now we will install the build dependencies and the kernel source itself:
sudo apt-get build-dep linux-image-$(uname -r)
sudo apt-get source linux-image-$(uname -r)

This should create a folder and a bunch of files in the current folder. My folder is called linux-hwe-4.8.0:
loli@ubongo:~$ ls -lt
total 671844
drwxr-xr-x 30 root root      4096 mar 12 16:46 linux-hwe-4.8.0
...
-rw-r--r--  1 root root      8543 mar  7 18:45 linux-hwe_4.8.0-41.44~16.04.1.dsc
-rw-r--r--  1 root root  13967968 mar  7 18:45 linux-hwe_4.8.0-41.44~16.04.1.diff.gz
-rw-r--r--  1 root root 139977270 feb  3 22:48 linux-hwe_4.8.0.orig.tar.gz
...

Since it's owned by root, we need to chown it to our user (replace loli with your username):
sudo chown -R loli:loli linux-hwe*

Time to edit the kernel source!
To find the quirks code, you can search for your GPU's model codes, but if you can't figure it out, chances are that it'll be in the same place as mine and you can just skip ahead.
loli@ubongo:~$ dmesg | grep "kernel modesetting"
[    0.733766] [drm] amdgpu kernel modesetting enabled.
[    0.736594] [drm] initializing kernel modesetting (PITCAIRN 0x1002:0x6810 0x174B:0xE271 0x00).

0x1002 is the vendor id which will be likely aliased to a constant, while 0x6810 0x174B:0xE271 are numbers specific to our particular GPU model.
Let's search the kernel code for these three:
loli@ubongo:~$ cd linux-hwe-*
loli@ubongo:~/linux-hwe-4.8.0$ grep -rin "0x6810.*0x174B.*0xE271" .
./drivers/gpu/drm/radeon/si_dpm.c:2930:	{ PCI_VENDOR_ID_ATI, 0x6810, 0x174b, 0xe271, 85000, 90000 },

(if you don't find anything, carry on and in the second edit you might find something that can help your gpu)
Those 85000, 90000 match our current clock speeds! Let's see the full code by running gedit +2930 ./drivers/gpu/drm/radeon/si_dpm.c to jump straight to line 2930:
/* cards with dpm stability problems */
static struct si_dpm_quirk si_dpm_quirk_list[] = {
	/* PITCAIRN - https://bugs.freedesktop.org/show_bug.cgi?id=76490 */
	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1462, 0x3036, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0xe271, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6810, 0x174b, 0xe271, 85000, 90000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1462, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1043, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x148c, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1682, 0x9275, 0, 120000 },
	{ 0, 0, 0, 0 },
};
Yep! Seems like this specific model is being downclocked quite a bit due to reports on stability issues, but I can already tell you there were no stability issues for me at full clock speed.
Let's comment out that line:
/* cards with dpm stability problems */
static struct si_dpm_quirk si_dpm_quirk_list[] = {
	/* PITCAIRN - https://bugs.freedesktop.org/show_bug.cgi?id=76490 */
	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1462, 0x3036, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0xe271, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0x2015, 0, 120000 },
	/* { PCI_VENDOR_ID_ATI, 0x6810, 0x174b, 0xe271, 85000, 90000 }, */
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1462, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1043, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6811, 0x148c, 0x2015, 0, 120000 },
	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1682, 0x9275, 0, 120000 },
	{ 0, 0, 0, 0 },
};
I'm gonna save you some time and tell you that this will fix the core clock, but the memory will cap at 1200 MHz because another hardcoded limitation. If you navigate the code to where it uses si_dpm_quirk_list or simply search for "120000" you will find this code (line 3010 in my version of the radeon driver):
/* limit all SI kickers */
	if (rdev->family == CHIP_PITCAIRN) {
		if ((rdev->pdev->revision == 0x81) ||
		    (rdev->pdev->device == 0x6810) ||
		    (rdev->pdev->device == 0x6811) ||
		    (rdev->pdev->device == 0x6816) ||
		    (rdev->pdev->device == 0x6817) ||
		    (rdev->pdev->device == 0x6806))
			max_mclk = 120000;
Change that to your card's maximum memory frequency multiplied by 100 (mine is flashed to 1450 MHz):
/* limit all SI kickers */
	if (rdev->family == CHIP_PITCAIRN) {
		if ((rdev->pdev->revision == 0x81) ||
		    (rdev->pdev->device == 0x6810) ||
		    (rdev->pdev->device == 0x6811) ||
		    (rdev->pdev->device == 0x6816) ||
		    (rdev->pdev->device == 0x6817) ||
		    (rdev->pdev->device == 0x6806))
			max_mclk = 145000;
Obviously if you don't have a PITCAIRN chip you need to edit the proper block of code for your specific gpu. Note that you can also simple comment out "(rdev->pdev->device == 0x6810) ||" from the if statement instead of editing the max clock (or whichever has your model number).
If you couldn't find any quirks for your gpu model earlier, take a good look at this code and see if you can find anything that caps clock speeds for your gpu. For example, I can see it setting max mem and core clock for VERDE, OLAND and HAINAN chips.
Now we can recompile the kernel (replace 9 with the number of threads + 1 in your cpu for faster compilation).
fakeroot debian/rules clean
CONCURRENCY_LEVEL=9 AUTOBUILD=1 fakeroot debian/rules binary-generic

This will take quite a while, probably at least 20 minutes even on a good CPU, as the ubuntu kernel comes with a massive amount of features. On gentoo I can compile the kernel much much faster than this.
And here's all our compiled debs!
loli@ubongo:~/linux-hwe-4.8.0$ ls -l ../*.deb
-rw-r--r-- 1 loli loli   214214 mar 12 18:49 ../linux-cloud-tools-4.8.0-41-generic_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli  1013226 mar 12 18:49 ../linux-headers-4.8.0-41-generic_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli   228130 mar 12 17:53 ../linux-hwe-cloud-tools-4.8.0-41_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli   905920 mar 12 17:53 ../linux-hwe-tools-4.8.0-41_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli 23703502 mar 12 18:49 ../linux-image-4.8.0-41-generic_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli 37895848 mar 12 18:49 ../linux-image-extra-4.8.0-41-generic_4.8.0-41.44~16.04.1_amd64.deb
-rw-r--r-- 1 loli loli   214268 mar 12 18:49 ../linux-tools-4.8.0-41-generic_4.8.0-41.44~16.04.1_amd64.deb

Before we proceed, make sure that in case anything goes wrong, we still get text mode:
echo vesafb | sudo tee -a /etc/initramfs-tools/modules
echo fbcon | sudo tee -a /etc/initramfs-tools/modules

Now install the kernel:
sudo dpkg -i ../linux-image-*_amd64.deb
sudo dpkg -i ../linux-headers-*_amd64.deb

As of now, ubuntu automatically updates initramfs and grub when you install the kernel debs so we should be ready to boot.
Reboot into the new kernel and verify that your clock speeds are indeed as expected (I flashed my r9 270x with an overclocked bios, stock clocks would be 107000 core and 1228 or so vddc):
(replace "radeon" with "amdgpu if you're using amdgpu)
loli@ubongo:~$ sudo su
[sudo] password for loli: 
root@ubongo:/home/loli# echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level
root@ubongo:/home/loli# cat /sys/kernel/debug/dri/0/radeon_pm_info
uvd    vclk: 0 dclk: 0
power level 3    sclk: 117500 mclk: 140000 vddc: 1300 vddci: 1025 pcie gen: 3
root@ubongo:/home/loli# echo auto > /sys/class/drm/card0/device/power_dpm_force_performance_level

For some reason it won't go higher than 1400 MHz even if I set the max_mclk higher in the code, but at least we're up to stock speed for the memory.
Uname should also show your custom kernel and the time it was built:
loli@ubongo:~$ uname -a
Linux ubongo 4.8.0-41-generic #44~16.04.1 SMP Sun Mar 12 18:24:17 CET 2017 x86_64 x86_64 x86_64 GNU/Linux

amdgpu-pro

Now, while this works for the nonproprietary radeon and amdgpu drivers, it won't work for amdgpu-pro, which is outside of the kernel tree. In that case you need to tweak the source for the amdgpu-pro module, which in my case was in /usr/src/amdgpu-pro and had the same exact quirk:
loli@ubongo:~/linux-hwe-4.8.0$ grep -rin "0x6810.*0x174B.*0xE271" /usr/src/amdgpu-pro*
/usr/src/amdgpu-pro-16.60-379184/amd/amdgpu/si_dpm.c:3026:	{ PCI_VENDOR_ID_ATI, 0x6810, 0x174b, 0xe271, 85000, 90000 },

loli@ubongo:~/linux-hwe-4.8.0$ grep -in "120000" /usr/src/amdgpu-pro-16.60-379184/amd/amdgpu/si_dpm.c
3023:	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1462, 0x3036, 0, 120000 },
3024:	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0xe271, 0, 120000 },
3025:	{ PCI_VENDOR_ID_ATI, 0x6811, 0x174b, 0x2015, 0, 120000 },
3027:	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1462, 0x2015, 0, 120000 },
3028:	{ PCI_VENDOR_ID_ATI, 0x6811, 0x1043, 0x2015, 0, 120000 },
3029:	{ PCI_VENDOR_ID_ATI, 0x6811, 0x148c, 0x2015, 0, 120000 },
3030:	{ PCI_VENDOR_ID_ATI, 0x6810, 0x1682, 0x9275, 0, 120000 },
3489:			max_mclk = 120000;

Edit it just like with the radeon driver (except you need to sudo gedit because the src directory is root) and then you can rebuild and reinstall the module.
sudo dkms remove amdgpu-pro/16.60-379184 --all
sudo dkms add amdgpu-pro/16.60-379184
sudo dkms build amdgpu-pro/16.60-379184
sudo dkms install amdgpu-pro/16.60-379184

Reboot and check your clocks:
loli@ubongo:~$ sudo su
[sudo] password for loli: 
root@ubongo:/home/loli# echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level
root@ubongo:/home/loli# cat /sys/kernel/debug/dri/0/amdgpu_pm_info
uvd    vclk: 0 dclk: 0
power level 3    sclk: 117500 mclk: 140000 vddc: 1300 vddci: 1025 pcie gen: 3
root@ubongo:/home/loli# echo auto > /sys/class/drm/card0/device/power_dpm_force_performance_level