napobear

Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing

this gist is part of this series

This assumes you are running Proxmox 8.2 and that the line source /etc/network/interfaces.d/* is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).

This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)

This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

napobear

Thunderbolt Networking

this gist is part of this series

NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve OR HIGER

This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol)

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

napobear

dont buy ASUS motherboard for this topology - many of their motherboard do not implement full SW Connection Manager required for full spec USB4 / TB4 required for this topology and they are unable to articulate which motherboard do and do not have the required features and keep saying 20Gbp/s is the max speed of connection on intel NUCS - this is wrong. 13th and 14th Gen NUCs that implement a SW connection manager for USB4 / TB4 will work fine..

ProxMox Cluster - Soup-to-Nutz

aka what i did to get from nothing to done.

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust, some items, less so). YMMV

Purpose of Proxmox cluster project

Required Outomces of cluster project

napobear

First raw Draft

echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-ns.list
echo "deb http://download.proxmox.com/debian/ceph-squid bookworm no-subscription" > /etc/apt/sources.list.d/ceph-ns.list

echo "thunderbolt" >> /etc/modules
echo "thunderbolt-net" >> /etc/modules

apt install vim lldpd thunderbolt-tools frr iperf3

napobear

Shopping List

Minisforum MS-01 Barebones

I was looking for mini PCs with SFP+ and found a lot of fairly expensive small servers that were tempting. Then I got lucky and saw a new product coming out from minisforum, the MS-01, which had everything I needed at a much lower pricepoint.

I went with the 20 core intel i9-13900H but I think any of the three would have been fine for my needs.

napobear

Package needed to run eclipse-installer on openSUSE Tumbleweed

sudo zypper in libgthread-2_0-0

napobear

Emulating Raspberry Pi 4 with Qemu

Just a quick update before we dive in: what we're actually doing here is running Raspberry Pi OS (64-bit) on a QEMU virtual ARM setup. This isn't full-blown hardware emulation of the Raspberry Pi 4, but more about creating a virtual environment for the OS. It doesn't mimic all the specific hardware features of the Pi 4, but it's pretty useful and great for general testing. I turned to this solution mainly to extract a modified sysroot from the Raspberry Pi OS, something not readily available in other resources. For those looking into detailed emulation of the actual Raspberry Pi 4's hardware in QEMU, check out this link for the latest updates: https://gitlab.com/qemu-project/qemu/-/issues/1208.

Hope it helps! :D

Shortcomings: No GUI yet, only console.

Steps

napobear

A Short IPv6 Guide for Home IPv4 Admins

This guide is for homelab admins who understand IPv4s well but find setting up IPv6 hard or annoying because things work differently. In some ways, managing an IPv6 network can be simpler than IPv4, one just needs to learn some new concepts and discard some old ones.

Let’s begin.

First of all, there are some concepts that one must unlearn from ipv4:

Concept 1

napobear

#!/bin/sh
figlist | awk '/fonts/ {f=1;next} /control/ {f=0} f {print}' | while read font; do
  echo "=== $font ==="
  echo $font | figlet -f $font
done | less

napobear

Routing plex traffic through an SSH tunnel

This guide creates a reverse SSH tunnel to route all Plex server traffic through it.

Step 2 is done on the tunnel, all other steps are done on the plex server.

1. Setup SSH keys (if you already have key based authenthication setup skip to step 2)

On plex server: