Skip to content

Instantly share code, notes, and snippets.

@SQLJames
Last active March 29, 2024 09:01
Show Gist options
  • Star 11 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save SQLJames/fe6fcd5e819d864986ce2eff6ad350da to your computer and use it in GitHub Desktop.
Save SQLJames/fe6fcd5e819d864986ce2eff6ad350da to your computer and use it in GitHub Desktop.
proxmox realtek driver issues

Edit 1

Updating the proxmox kernel will require you to remove this and potentially reapply https://forum.proxmox.com/threads/autoinstall-for-kernel-6-5-11-3-pve-failed.136818/

Ansible role

https://github.com/PragmaticEngineering/ansibleRole-proxmox-realtek

Descripton

As part of proxmox 8.0 the realtek drivers are having issues where after a certain amount of time the node will go offline and lose connectivity. Here are the steps I took with a no-subscription repo.

to know if you could be affected by this issue, log into your proxmox host, either via ssh or keyboard and display and run:

lspci -nnk

if you are affected, you should see a Ethernet controller from Realtek Semiconductor Co. using the r8169 driver

for example

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:07a3]
        Kernel driver in use: r8169
        Kernel modules: r8169

Summary

vi /etc/apt/sources.list


deb http://ftp.us.debian.org/debian bookworm main contrib non-free non-free-firmware

deb http://ftp.us.debian.org/debian bookworm-updates main contrib non-free non-free-firmware

# security updates
deb http://security.debian.org bookworm-security main contrib non-free non-free-firmware

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

apt update

apt install pve-headers -y

apt install r8168-dkms -y

apt upgrade -y

echo blacklist r8169 >> /etc/modprobe.d/blacklist-r8169.conf

sudo sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3"/' /etc/default/grub

proxmox-boot-tool refresh

reboot -n

Walkthrough

modify the repository source list. These repositories provide packages and updates for your system. vi /etc/apt/sources.list

deb http://ftp.us.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://ftp.us.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
# security updates
deb http://security.debian.org bookworm-security main contrib non-free non-free-firmware
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

After modifying the sources.list file we need to make sure the system updates the package list from the repositories

apt update

Heres where most issues came up for me, we need to install the pve-headers pacakge. because without it, we will have issues compiling the realtek-dkms package. The -dkms suffix indicates that it's a Dynamic Kernel Module Support package, which means it can be automatically rebuilt whenever the kernel is updated. When you install this package, it uses the kernel headers (provided by pve-headers) to build and install the Realtek network card driver as a kernel module.

apt install pve-headers -y

Install the Realtek 8168 network card driver as a Dynamic Kernel Module Support package. It can automatically rebuild the driver when the kernel is updated.

apt install r8168-dkms -y

Then, for good measure I made sure my other packages were updated.

apt upgrade -y

blacklist the 8169 driver from being loaded

echo blacklist r8169 >> /etc/modprobe.d/blacklist-r8169.conf

Once all of this is completed we are going to update our GRUB to have some specific kernel parameters for our realtek driver

sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3"/' /etc/default/grub

Finally, we are are going to refresh the boot configuration for Proxmox and reboot.

proxmox-boot-tool refresh reboot -n

@downtime0
Copy link

downtime0 commented Sep 27, 2023

Thank you, Thank you. I found this from your youtube video.
This fixed my new N100 with duel nics (1 Gbe and 2.5 Gbe). But interesting it only changed the 1 Gbe RTL8168, while the RTL8125 (2.5 Gbe) was unaffected and continues to use the r8169 driver even after it was blacklisted. Only the 1 Gbe nic was ever a problem.
I freshly installed Proxmox 8.0.4, performed this fix, and moved my pfSense, pihole, and Unifi Controller VMs over to this and am getting my full 800 Mbs down on speedtest.net.
realtek screenshot 2023-09-27

@mickb2023
Copy link

Thank you so much, After getting the R8168 drive working on my asus prime n100 with proxmox I could not get the speed to go over 350mbs spent ages trying things. Booted open suse today and it worked fine. So was thinking I would need to wait for a new kernel. Bur after adding your kernel options its working perfectly. You sir are a gent. Thank you so so much

@CrayzChris
Copy link

CrayzChris commented Sep 29, 2023

Thanks SQLJames for you work!

You have a idea why its not working on my Optiplex 3000 Thin Clients? :(

lspci show me

        Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:0ae8]
        Kernel driver in use: r8169
        Kernel modules: r8169

after install driver

        Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:0ae8]
        Kernel driver in use: r8168
        Kernel modules: r816````

but the performance is bad too maybe ~ 500KB/s 

@SQLJames
Copy link
Author

SQLJames commented Sep 30, 2023

when you are looking at the performance of the machine, how are you testing it? I am running multiple dell 3080 mini pcs for my proxmox setup and I was able to do a iperf3 test on my local network and they got normal performance for a gigabit nic.

This was run from a raspberry pi to the proxmox host. (not internal to the proxmox cluster)

root@proxmox-01:~# iperf3 -c 172.16.32.1
Connecting to host 172.16.32.1, port 5201
[  5] local 172.16.12.33 port 54278 connected to 172.16.32.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   947 Mbits/sec    0    259 KBytes
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    0    259 KBytes
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    259 KBytes
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    0    259 KBytes
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    259 KBytes
[  5]   5.00-6.00   sec   113 MBytes   944 Mbits/sec    0    272 KBytes
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    291 KBytes
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec    0    291 KBytes
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    291 KBytes
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    291 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.10 GBytes   938 Mbits/sec                  receiver
root@proxmox-01:~# lsblk -nnk
... 
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
        Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1028:09a8]
        Kernel driver in use: r8168
        Kernel modules: r8168

for transparency, here is the information from the server side.

iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.16.12.33, port 54272
[  5] local 172.16.32.1 port 5201 connected to 172.16.12.33 port 54278
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   109 MBytes   911 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.03  sec  3.27 MBytes   938 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.03  sec  1.10 GBytes   938 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

@rcv11x
Copy link

rcv11x commented Oct 3, 2023

hi @SQLJames, I am using a pc as a server that has a msi b660m pro am4 board and an i5 12500 with a realtek 2. 5gbps but look when I run the command lspci -nnk both before and after running all commands still appears the r8169 driver I have been using the server for 1 year and never gave me to look at this, I have contracted 1gbit at home and when I do a speedtest pro console on the host proxmox or a lxc only get 600mbits instead in a virtual machine gives me everything perfect that are about 920mbit, because it can be?

apart from this next week I get a mini pc called FIREBAT-Mini PC T8 PRO PLUS that has an intel n100 and 2 ports 1 gbit that are realtek, I hope this guide will help me with that, but leaving that aside because this happens to me on my pc?

in the speedtest i am connected with a vpn wireguard in another place that is not the server but it doesn't matter because i am running the test inside proxmox from the interface so it should give me at least 800mbits.
Screenshot_20231003_112540

Speedtest result in proxmox host panel:
Screenshot_20231003_113822

@Kh3nsu
Copy link

Kh3nsu commented Oct 11, 2023

I have just updated the driver like in your manual, the kernel module changed to r8168 but it still says vfio-pci like before.

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
DeviceName: Onboard LAN Brodcom
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:0123]
Kernel driver in use: vfio-pci
Kernel modules: r8168

I did this as I also suffer from the problem, not sure if I still do after this update. I'll report back

Edit (15.10.23): Seems to work, I had no disconnects since I installed the older driver. Thanks!!!

@petersjostrom
Copy link

This doesn't work on Proxmox 8 for me, I get this error on loading r8168-dkms:
module verification failed: signature and/or required key missing - tainting kernel

Before I go down the rabbit hole of trying to rebuild the kernel etc, is there any easier way to solve this?

@Kh3nsu
Copy link

Kh3nsu commented Oct 15, 2023

This doesn't work on Proxmox 8 for me, I get this error on loading r8168-dkms: module verification failed: signature and/or required key missing - tainting kernel

Before I go down the rabbit hole of trying to rebuild the kernel etc, is there any easier way to solve this?

What version of Proxmox 8 are you running on?
I'm running on 8.0.4 (Linux pve 6.2.16-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-15 (2023-09-28T13:53Z) x86_64 GNU/Linux)
and I installed it perfectly fine without any issues. It also solved my problem.
Just checked my dmesg log and I also get this error: ("dmesg | grep r8168")

[    1.075929] r8168: module verification failed: signature and/or required key missing - tainting kernel
[    1.078263] r8168 Gigabit Ethernet driver 8.051.02-NAPI loaded

Which can be ignored as the driver is still being loaded as you can see in the line after. This "verification" error is just an information for you that you are using a kernel which is not fully verified or tested.
Please check the installed driver / kernel module of your adapter with:

lspci -nnk

I'm afraid there probably is no other solution until Proxmox updates the driver in the kernel by themselves.

@petersjostrom
Copy link

petersjostrom commented Oct 15, 2023

I'm using Proxmox 8.0.4, upgraded all nodes yesterday.

Aha, sorry, you were right so far. But it itsn't loading r8168 for the NIC, it still manages to load r8169 for the NIC, but initially it didn't get any IP, I still couldn't ping it from outside. Turns out the links were DOWN and I could "ifup enp2s0" and "ifup vmbr0" to get them running on r8169 again. So the blacklist of r8169 isn't working?

"lsmod | grep r81" says I have both r8169 and r8168.
"ethtool -i enp2s0" says it's using the r8169 driver.

I have three files in /etc/modprobe.d/, after trying multiple times now and before upgrading to Proxmox 8, I've tried to make it clear, but still no go:

pve-blacklist.conf:
blacklist nvidiafb

dkms.conf:

r8168-dkms.conf:
blacklist r8169

Still, the NIC uses r8169. What's wrong, why isn't blacklist working?

@shaunsull
Copy link

shaunsull commented Oct 21, 2023

Thank you @SQLJames . I was going crazy troubleshooting my network to find out why one of my nodes was taking me back to dial-up modem days despite being the newest hardware in my homelab. This solved the issue perfectly for the 1Gbe NIC. I also have result as @petersjostrom with the 2.5 Gbe NIC's reporting r8169.

root@02-node-msi:~# lsmod | grep r81
r8169                 114688  0
r8168                 655360  0
root@02-node-msi:~# ethtool -i enp45s0
driver: r8169
version: 6.2.16-15-pve
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version: 
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

root@02-node-msi:~# ethtool -i enp46s0
driver: r8168
version: 8.051.02-NAPI
firmware-version: 
expansion-rom-version: 
bus-info: 0000:2e:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
root@02-node-msi:~# 

But with my primary NIC working at full speed I am so happy to have found this post.

@woody4165
Copy link

Hi

in my /etc/apt/sources.list I have bullseye instead of bookworm, like this

deb http://ftp.debian.org/debian bullseye main contrib non-free non-free-firmware
deb http://ftp.debian.org/debian bullseye-updates main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security bullseye-security main contrib non-free non-free-firmware
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

#deb http://ftp.debian.org/debian bullseye main contrib
#deb http://ftp.debian.org/debian bullseye-updates main contrib
#deb http://security.debian.org/debian-security bullseye-security main contrib

My Proxmox is

root@pve:/etc# pveversion
pve-manager/7.4-17/513c62be (running kernel: 6.2.16-11-bpo11-pve)

Can I leave bullseye or should I change to bookworm?
What could be the impact on my actual Proxmox configuration?

If i do an apt list command searching for r8168, I found this

r8168-dkms/oldstable 8.048.03-3 all

Is this the one I have to install ?

Thanks

@Aussiefox
Copy link

Thanks SQLJames, working well. As a Linux noob, does this need to be undone at some point?

@encg
Copy link

encg commented Nov 18, 2023

Thanks @SQLJames! My OptiPlex 3080 was having this exact problem.

@snod
Copy link

snod commented Nov 21, 2023

FIREBAT-Mini PC T8 PRO PLUS

I can confirm that this approach works on said device.

@lupsys
Copy link

lupsys commented Nov 25, 2023

After following the steps, when I restart, the card doesn't start automatically. I have to start it manually with modprobe, then bring up the card, and finally restart with systemctl restart networking. What could be happening? I've followed the steps exactly, copy and paste.
After installation, I will check if the module has been installed in /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek, and if the r8168.ko module is not present, only r8169.ko. This is normal because it does not load automatically every time I restart.

@shaunsull
Copy link

See this forum post for the 6.5.11-4-pve kernel. update. I know nothing about how all this driver issue works, but this fixed my issue after upgrading. Since I upgraded three nodes without reading this first (duh!), I had to connect a keyboard and monitor and boot to the old kernel. Note to self, wait for updates to settle and check the forums before jumping in.

https://forum.proxmox.com/threads/proxmox-ve-8-1-released.136960/post-608898

To be able to update the kernel to 6.5.11-4-pve, WITH R8168 kernel module working, you need to build the kernel module for the new version of the kernel. The module was previously build for the kernel version you were running.

@PP-Mclappins
Copy link

I've done the fix, and I am now seeing the correct driver in place when running "lspci -nnk"

Unfortunately after the node reboot, I am still not seeing the card in my networking tab as a Network interface.

going to attempt to manually bring the card up, but not sure how it's going to go, Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment