AGitUser111

Setup HA Windows VM

this example uses windows server 2022. Windows 11 will be basically the same

It is beyond this gist to be a deep tutorial.

this gist is part of this series

Prep

1. Download your windows ISO of choice (i am using windows 2022 Data Center for this example.)
2. Download the lates virtio.iso too

AGitUser111

Configuring CephFS to store ISOs for VMs and Templates for Containers

This gist assume you have setup Ceph as per earlier gist in this sequence

I found it annoying to upload ISO to each node in the cluster, download CT templates to each node.
I thought setting up cedph for these would be a great idea (and it is).

this gist is part of this series

create cephfs pools

1. In the gui navigate to Datcenter > node name > ceph

AGitUser111

CEPH HA Setup

Note this should only be done once you are sure you have reliable TB mesh network.

this is because proxmox UI seems fragile wrt to changing underlying network after configuration of ceph.

All installation done via command line due to gui not understanding the mesh network

This setup doesn't attempt to seperate the ceph public network and ceph cluster network (not same as proxmox clutser network), The goal is to get an easy working setup.

this gist is part of this series

AGitUser111

Proxmox cluster Setup

this gist is part of this series

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

AGitUser111

Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing

This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

Note for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....

this gist is part of this series

Create Loopback interfaces

Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.

AGitUser111

Thunderbolt Networking

this gist is part of this series

NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve OR HIGER

This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol)

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

AGitUser111

Hardware Setup / Proxmox Base Install

this gist is part of this series

Hardware Setup (perform on all 3 nodes)

• Connect all 3 NUC to power etc
• Disable secure boot in BIOS on all nodes
• Connect all 3 TB4 cables in a rinng topology:

using the numbers printed on the case of the intel13 nucs connect cables as follows (this is important):

AGitUser111

ProxMox Cluster - Soup-to-Nutz

aka what i did to get from nothing to done.

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust, some items, less so). YMMV

Purpose of Proxmox cluster project

Required Outomces of cluster project

AGitUser111

Shopping List

Minisforum MS-01 Barebones

I was looking for mini PCs with SFP+ and found a lot of fairly expensive small servers that were tempting. Then I got lucky and saw a new product coming out from minisforum, the MS-01, which had everything I needed at a much lower pricepoint.

I went with the 20 core intel i9-13900H but I think any of the three would have been fine for my needs.