OhGodAGirl/a file.md

## a file.md

      
    Raw
  

              a file.md
            
          
    Simple guide to reverse engineer Nvidia GPUs

Known TODOs

Trello https://trello.com/b/ZudRDiTL/nouveau

The Nouveau Trello board contains a list of known things which still needs to be implemented or fixed.
Freedesktop Bugzilla

Here there are mainly two kinds of issues tracked
kernel https://bugs.freedesktop.org/buglist.cgi?component=Driver%2Fnouveau&list_id=600638&product=xorg&resolution=---

All bugs related to the kernel modules are found here
mesa: https://bugs.freedesktop.org/buglist.cgi?component=Drivers%2FDRI%2Fnouveau&list_id=600639&product=Mesa&resolution=---

All bugs related to the userspace parts are found here
General stuff

Terms

POST

see https://en.wikipedia.org/wiki/Power-on_self-test
Tools

nvalist

Nvalist prints a list of all available devices, which can be used to identify, which number has to be passed to the -c flags of most nva tools:
$index: more information
$index is the number to use for the -c flag
Example output

0: (pci) 0000:02:00.0 MCP79 0ac080b1
1: (pci) 0000:00:03.5 SMU
2: (pci) 0000:00:0a.0 ETH

MMIO registers

The MMIO registers are used to communicate with the actual hardware. Envytools contains a database with all known registers.
General Tools

nvapeek register [range]

Reads out the given MMIO register (range).
nvapoke register [range] value

Writes the given value into the given register (range).
rnndb

This is a XML based databse with all known MMIO registers. Files are located in envytools.git/rnndb and should be extended whenever something new is found out.
nvaforcetemp temp

Can be used to force the GPU to return the given temperature on MMIO register read outs. 0 to disable the override.
NOTICE: Due to secure boot, won't work on GM200+ GPUs
Nvidia VBIOS

Contains the manufacturer-provided GPU model specifications. Nvbios can be used to parse parts of the vbios, which were already reverse engineered. Look there for information related to power management, connectors, clocks, memory, init scripts, gpios, external devices and many other things.
WARNING: Some data may depend on the value of the strap MMIO register (0x101000), which allow manufacturers to share the same vbios with multiple revisions of a GPU. Information such as the memory information may be indexed based on this value, check out the uses of the strap_peek in nvbios for more information.
INFO: Nouveau developers have access to a repository of user vbioses, ask for it if you feel like you need it.
Extracting the vbios

Running Nouveau

While running nouveau, the vbios can be extracted through debugfs quite easily. The file is located at /sys/kernel/debug/dri/x/vbios.rom, where x represant the drm id of the device.
nvagetbios -s prom

Can be used to extract the vbios from the device without requiring any drivers. Not as reliable as using Nouveau, because nvagetbios only supports prom (on chip vbios) and pramin (vbios in vram). Doesn't support ACPI embedded vbios.
Tools

nvbios [-c cardnum] [-v] vbios.rom

parses the given vbios.rom file. The -v flag also prints the actual hex values under the parsed rows.
nvafakebios [-eEwWlL reg:val] file

Uploads the given vbios file on the GPU. The GPU has to be POSTed, otherwise random memory corruption will occur and most likely the system will crash. The easiest way to POST the card is to load any video driver for it and unload it again, preferable nvidia.
If the 0x619f04 MMIO register isn't set to a sane value after POSTing the card (bytes 32-8 are 0), it needs to be adjusted with nvapoke according to this formula:
((VRAM size * 0x10 - 2) * 0x100) | 0x9
example with 3072MB vram:
((3072 * 0x10 - 2) * 0x100) | 0x9 = 0xbffe09 means you have to nvapoke 0x619f04 0xbffe09
The -eEwWlL flags can be used to directly modify bytes within the vbios

e: 8bit hex
E: 8bit dec
w: 16bit hex
W: 16bit dec
l: 32bit hex
L: 32bit dec

example: nvafakebios -w 100:4ff vbios.rom will turn byte 0x100 into 0xff and 0x101 into 0x04
modprobe -r nvidia

Combined with the modprobe.d.nvidia-rmmod.conf file put into /etc/modprobe.d/nvidia-rmmod.conf provides an easy way to unload nvidia and all modules depending on it.
For this to work, nvidia has to be loaded through anything which parses modeprobe.d files like modprobe.
Nouveau Userspace Library

Nouveau can be compiled as a library from the top level directory of the Nouveau git repository. This library is usefull to write tools based on the current Nouveau code to get access to most of the subdevs from nvkm. See the Nouveau Kernel Module Documentation for more details.
With this it is possible to write a tool, which can be run alongside Nvidia to test Nouveau against Nvidia.
Be aware that Nouveau won't read the faked vbios.
any hex editor for modifying the vbios file

Actually it is possible to modify the vbios while uploading it with nvafakebios, but a HEX editor provides a more comfortable way to modify multiple bytes at once
Procedure


Adjust your vbios
run nvafakebios
load Nvidia, start X
collect data
unload Nvidia/reboot
repeat

It is recommended to write a script using envytools or a tool based on the Nouveau Userspace Library to have a reproduciable tests to verify and reproduce changes.
If the function of bits/bytes in the vbios isn't clear, try to collect enough data to be able to make data driven assumptions. They can be approved or disproved later on through the same means as above.
Be aware that Nvidia may fail to load if something important was changed.
Example script

#!/bin/sh
# reads out PWM0 and PWM1 state to detect changes (maybe fans are controlled through them?)
echo old state
nvapeek 0x200c4 0x18

echo lower temperature
nvaforcetemp 1
nvapeek 0x200c4 0x18
sleep 2 
nvapeek 0x200c4 0x18

raise temperature
nvaforcetemp 95
nvapeek 0x200c4 0x18
sleep 2
nvapeek 0x200c4 0x18

echo reset temperature
nvaforecetemp 0
Nvidia kernel module

initial set-up, power management
Tools

mmiotrace

A Linux kernel module, which traces register accesses inside the MMIO space by acting as a page fault handler.
More information can be found on the ubuntu wiki: https://wiki.ubuntu.com/X/MMIOTracing
Warning: Only limited support for repeate instructions, so consecutive reads/writes aren't cought.
demmio

demmio is used to make an mmiotrace more human readable by enhancing it with known register from rnndb and also parse embedded scripts like HWSQ or SEQ ones, which are used for memory reclocking.
Nvidia Userspace libraries

Introduction

User-space == Per-client work, command submission (link to pfifo's documentation), OpenGL, OpenCL, etc... is implemented there
Microcodes: NVIDIA is known to use a lot of different microcodes. You may use nvdis to de-assemble them
Tools

valgrind-mmt

demmt

Additional tweaks for an Optimus Setup

On Optimus setups it is possible to unload the nvidia/nouveau driver and even fully reset your GPU without the need of rebooting makeing reverse engineering much faster on such systems.
Tools

bbswitch

bbswitch is a kernel module used alongside bumblebee to turn on/off the GPU without the need of the actual GPU driver supporting this. Just echo "ON" or "OFF" into /proc/acpi/bbswitch.
"better" Xorg config

the xorg.conf.d.nouveau.conf file prevents the main X server of claiming the GPU with any driver. This way even if Nouveau gets loaded at runtime, Xorg won't claim a reference, so Nouveau can be unloaded even while X is still running.

  
## fuc_as.sh
#!/bin/sh
# usage: fuc_as path/to/dir/containing/fuc/files
for f in $(find "$1" -iname "*.fuc[345]");
        do echo $assembling $f
        cat $f | cpp -CC -I"$1" | cpp | sed -e '/^#/d'  | envyas -a -w -m falcon -V ${f##*.} > $f.h
done

## modprobe.d.nvidia-rmmod.conf
nvidia-rmmod.conf
# Nvidia
remove nvidia modprobe -r --ignore-remove nvidia-drm nvidia-modeset nvidia-uvm nvidia

## nvctrl.c
#include <X11/Xlib.h>

#include "NVCtrl.h"
#include "NVCtrlLib.h"
#include "NvCtrlAttributes.h"

int main() {
	Display *dpy;
	char * utilizationStr;
	const CtrlTarget * ctrl_target;
	int ret, display_devices;

	dpy = XOpenDisplay(NULL);

	ret = XNVCTRLQueryTargetStringAttribute(dpy,
		NV_CTRL_TARGET_TYPE_GPU,
		0, // target_id
		0, // display_mask
		NV_CTRL_STRING_GPU_UTILIZATION,
		&utilizationStr
	);

	if (!ret) {
		fprintf(stderr, "Failed to query connected displays\n");
		return 1;
	}

	printf("%s\n", utilizationStr);

	XCloseDisplay(dpy);
}

## xorg.conf.d.nouveau.conf
Section "ServerFlags"
        Option      "AutoAddGPU" "false"
        Option      "NoTrapSignals"
EndSection

Section "Device"
    Identifier "IntelCard"
    Driver "intel"
    Option "AccelMethod" "sna"
    Option "DRI" "3"
EndSection

Section "Device"
        Identifier "NvidiaCard"
        Driver "dummy"
        BusID "PCI:01:00:0" # PCI bus address of the Nvidia GPU here
EndSection
	#!/bin/sh
	# usage: fuc_as path/to/dir/containing/fuc/files
	for f in $(find "$1" -iname "*.fuc[345]");
	do echo $assembling $f
	cat $f \| cpp -CC -I"$1" \| cpp \| sed -e '/^#/d' \| envyas -a -w -m falcon -V ${f##*.} > $f.h
	done
	nvidia-rmmod.conf
	# Nvidia
	remove nvidia modprobe -r --ignore-remove nvidia-drm nvidia-modeset nvidia-uvm nvidia
	#include <X11/Xlib.h>

	#include "NVCtrl.h"
	#include "NVCtrlLib.h"
	#include "NvCtrlAttributes.h"

	int main() {
	Display *dpy;
	char * utilizationStr;
	const CtrlTarget * ctrl_target;
	int ret, display_devices;

	dpy = XOpenDisplay(NULL);

	ret = XNVCTRLQueryTargetStringAttribute(dpy,
	NV_CTRL_TARGET_TYPE_GPU,
	0, // target_id
	0, // display_mask
	NV_CTRL_STRING_GPU_UTILIZATION,
	&utilizationStr
	);

	if (!ret) {
	fprintf(stderr, "Failed to query connected displays\n");
	return 1;
	}

	printf("%s\n", utilizationStr);

	XCloseDisplay(dpy);
	}
	Section "ServerFlags"
	Option "AutoAddGPU" "false"
	Option "NoTrapSignals"
	EndSection

	Section "Device"
	Identifier "IntelCard"
	Driver "intel"
	Option "AccelMethod" "sna"
	Option "DRI" "3"
	EndSection

	Section "Device"
	Identifier "NvidiaCard"
	Driver "dummy"
	BusID "PCI:01:00:0" # PCI bus address of the Nvidia GPU here
	EndSection