Skip to content

Instantly share code, notes, and snippets.

@kafisatz

kafisatz/proxmox-tb-net.md

Forked from scyto/proxmox-tb-net.md
Created September 15, 2023 06:57
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kafisatz/043a2f61d50bacf613ffe2c48400d4fd to your computer and use it in GitHub Desktop.
Save kafisatz/043a2f61d50bacf613ffe2c48400d4fd to your computer and use it in GitHub Desktop.
Download ZIP
Thunderbolt Networking Setup
Raw
proxmox-tb-net.md

Thunderbolt Networking

Install LLDP - this is great to see what nodes can see which.

  • install lldpctl with apt install lldpd

Load Kernel Modules

  • add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
    1. nano /etc/modules add modules at bottom of file, one on each line
    2. save using x then y then enter

Prepare /etc/network/interfaces

doing this means we don't have to give each thunderbolt a manual IPv6 addrees and that these addresses stay constant no matter what Add the following to each node using nano /etc/network/interfaces

If you see any sections called thunderbolt0 or thunderbol1 delete them at this point.

Now add the following (note we will set IP addresses in the UI):

auto en05
iface en05 inet static
        mtu 4000

iface en05 inet6 static
        mtu 4000

auto en06
iface en06 inet static
        mtu 4000

iface en06 inet6 static
        mtu 4000

If you see any thunderbol sections delete them from the file before you save it.

Rename Thunderbolt Connections

This is needed as proxmox doesn't recognize the thunderbolt interface name. There are various methods to do this. This method was selected after trial and error because:

  • the thunderboltX naming is not fixed to a port (it seems to be based on sequence you plug the cables in)
  • the MAC address of the interfaces changes with most cable insertion and removale events

  1. use udevadm monitor command to find your device IDs when you insert and remove each TB4 cable. Yes you can use other ways to do this, i recommend this one as it is great way to understand what udev does - the command proved more useful to me than the syslog or lspci command for troublehsooting thunderbolt issues and behavious. In my case my two pci paths are 0000:00:0d.2and 0000:00:0d.3 if you bought the same hardware this will be the same on all 3 units. Don't assume your PCI device paths will be the same as mine.

  2. create a link file using nano /etc/systemd/network/00-thunderbolt0.link and enter the following content:

[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05
  1. create a second link file using nano /etc/systemd/network/00-thunderbolt1.link and enter the following content:
[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en06

Set Interfaces to UP on reboots and cable insertions

This section en sure that the interfaces will be brought up at boot or cable insertion with whatever settings are in /etc/network/interfaces - this shouldn't need to be done, it seems like a bug in the way thunderbolt networking is handled (i assume this is debian wide but haven't checked).

  1. create a udev rule to detect for cable insertion using nano /etc/udev/rules.d/10-tb-en.rules with the following content:
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"

  1. save the file

  2. create the first script referenced above using nano /usr/local/bin/pve-en05.sh and with the follwing content:

#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en05

save the file and then

  1. create the second script referenced above using nano /usr/local/bin/pve-en06.sh and with the follwing content:
#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en06

and save the file

  1. make both scripts executable with chmod +x /usr/local/bin/*.sh
  2. Reboot (restarting networking, init 1 and init 3 are not good enough, so reboot)

Set IP addresses via UI

  1. At this point you can now use the proxmox ui (Datacenter > nodename > network to set the IP addresses in CIDR notation.
  2. Click Apply Configuration
  3. perform ip addr in the entries you should see something like this (assuming you set IPv4 and IPv6 - note my IPv6 network address is redacted below with Xs):
6: en05: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:e5:4a:ad:e4:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.81/24 scope global en05
       valid_lft forever preferred_lft forever
    inet6 xxxx:xxxx:xxxx:50::81/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::e5:4aff:fead:e464/64 scope link 
       valid_lft forever preferred_lft forever
7: en06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:46:9f:8e:35:16 brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.81/24 scope global en06
       valid_lft forever preferred_lft forever
    inet6 xxxx:xxxx:xxxxx:60::81/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::46:9fff:fe8e:3516/64 scope link 
       valid_lft forever preferred_lft forever

If not check all your steps in this section VERY carefully (oh and that your cables are connected)

*Ok now you your base networking should be in a good state, note you still have no IP connectivity over the TB links, that comes next.s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment