kafisatz

Thunderbolt Networking

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

Load Kernel Modules

• add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
  1. nano /etc/modules add modules at bottom of file, one on each line
  2. save using x then y then enter

kafisatz

PROXMOX Setup Weirdness Observed

1. Most of the time gui setup didn't get dhcp address from my server, one time in 6 installs it did.
2. Sometimes gui setup populated an example IPv4 address, sometimes an actual IPv6 address and one time real DCHP IPv4 address
3. on one node in 6 installs it added a thunderbol0 entry to /etc/network/interaces resulting in what you see below, the fix was to remove from interfaces file

4. sometimes after setup the tunderbolt modules load and sometimes they don't - i don't know why, i currently have 1 machine with them specified in /etc/modules file and 2 machines without - i figured out this is (stupidly) by design, one machine in the cluster needs to manually have the the modules loaded (but this will cause other machines to load them on the fly) as such its best to define them on all nodes.

5. on node 3 i accidentally gave

kafisatz

How To migrate Gen2 Windows VM from Hyper-V

This assumes you have Windows Server 2022 Gen2 VM running on hyper-v that uses gen2 with UEFI and Secure Boot - it should work for win11 but i haven't tested that. This has only tested with a vanilla windows server 2022 VM so far (three times to write guide). I will comment this gist when i manage to move one of my domain controllers.

VM preparation

1. Dowload Virtio DVD and attach to running windows VM on Hyper-v
2. Log onto console of Windows VM to be migrated
3. Install VirtIO drivers (all drivers, this will also include spice tools)
4. Gracefully shutdown server
5. Export VM to CIFS location accessible by proxmox and mounted in proxmox

kafisatz

Setup HA Windows VM

this example uses windows server 2022. Windows 11 will be basically the same

It is beyond this gist to be a deep tutorial.

Prep

1. Download your windows ISO of choice (i am using windows 2022 Data Center for this example.)
2. Download the lates virtio.iso too
3. go to Datacenter > pve1 > expand child nodes in left pane > local (pve1) (i don't know why i find this unintutive)
4. click ISO Images

kafisatz

Configuring CephFS to store ISOs for VMs and Templates for Containers

This gist assume you have setup Ceph as per earlier gist in this sequence

I found it annoying to upload ISO to each node in the cluster, download CT templates to each node.
I thought setting up cedph for these would be a great idea (and it is).

create cephfs pools

1. In the gui navigate to Datcenter > node name > ceph
2. Click cephfs node
3. In Metadata Server area click create and create a meta data server on all 3 pve nodes

kafisatz

CEPH HA Setup

Note this should only be done once you are sure you have reliable network. proxmox and ceph seem fragile wrt to changing underlying network after configuration of ceph or even during all installation done via command line due to gui not understanding the mesh network

This setup doesn't attempt to seperate the ceph public network and ceph cluster network (not same as proxmox clutser network), The goal is to get an easy working setup.

Ceph Initial Install & monitor creation

1. On all nodes execute the command pveceph install --repository no-subscription accept all the packages and install
2. On node 1 execute the command pveceph init --network 10.0.0.81/24

kafisatz

Proxmox cluster Setup

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

I have 3 networks:

kafisatz

Enable IPv4 OSPF Routing on Thunderbolt Mesh

This will result in a routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

Key Parameters

Key Information Used Note i used the 10.x IPv4 space as this is not used anywhere else on my network YMMV

Node l:

• lo:0 = 10.0.0.81/32

kafisatz

Hardware Setup / Proxmox Base Install

Hardware Setup (perform on all 3 nodes)

• Connect all 3 NUC to power etc
• Disable secure boot in BIOS on all nodes
• Connect all 3 TB4 cables in a rinng topology:

using the numbers printed on the case of the intel13 nucs connect cables as follows (this is important):
- node 1 port 1 > node 2 port 2
- node 2 port 1 > node 3 port 2

kafisatz

ProxMox Cluster Proof Of Concept (Work In Progress)

Start Date - First week of August 2023

End Date - none

Last Updated 9/1/2023

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust (some of the extra credit items, less so). YMMV i wiill regularly update until this become production