Skip to content

Instantly share code, notes, and snippets.

View Drallas's full-sized avatar
🎈

Drallas

🎈
  • Planet Earth
  • 11:32 (UTC +02:00)
View GitHub Profile
@Drallas
Drallas / Managing and Troubleshooting.md
Last active May 13, 2024 01:11
Configure Ceph & Troubleshooting

Proxmox High Available cluster with Ceph and Dynamic Routing - Managing and Troubleshooting

Part of collection: Hyper-converged Homelab with Proxmox

This is part 3 focussing on Managing and Troubleshooting Proxmox and Ceph.

See also Part 1 about Setup Networking for a High Available cluster with Ceph, and see Part 2 for how to setup the Proxmox and Ceph Cluster itself.

WIP

@Drallas
Drallas / High Available Pi-hole failover cluster using Keepalived and Orbital Sync.md
Last active May 13, 2024 01:11
High Available Pi-hole failover cluster using Keepalived and Orbital Sync

Docker Swarm in Vm's with CephFS

Part of collection: Hyper-converged Homelab with Proxmox

One of the objectives, of building my Proxmox HA Cluster, was to store persistent Docker volume data inside CephFS folders.

There are many different options to achieve this; via Docker Swarm in LXC using Bind Mounts, Docker Third Party Plugins that are hard to use and often outdated.

Another option for Docker Volumes was running GlusterFS, storing the disks on local NVMe storage and not using CephFS. Although appealing, it's adding complexity and unnecessary resource consumption, while I already have a High Available File System (CephFS) running!

Build a Proxmox High Available cluster with Ceph

Part of collection: Hyper-converged Homelab with Proxmox

This is part 2 focussing on building the Proxmox Cluster and setting up Ceph.

See also Part 1 about Setup Networking for a High Available cluster with Ceph, and see Part 2 for how to setup the Proxmox and Ceph Cluster itself, and part 3 focussing on Managing and Troubleshooting Proxmox and Ceph.

If everything went well in part 1, setting up Proxmox and Ceph should be 'a walk in the park'!

Build a Hyper-converged Proxmox HA Cluster with Ceph

Part of collection: Hyper-converged Homelab with Proxmox

This is part 1 focussing on the networking part of building a Proxmox High Available cluster with Ceph.

Part 2 focusses on building the Proxmox Cluster and setting up Ceph itself, and part 3 focussing on Managing and Troubleshooting Proxmox and Ceph.

Why

For some time, I was looking for options to build a Hyper-converged (HCI) Homelab, considering options like TrueNAS Scale, SUSE Harvester) among other option.

Hyper-converged High Available Homelab with Proxmox

This is me documenting my journey moving my Homelab from a Qnap NAS and a Single host Proxmox server to a Hyper-converged multi-node Proxmox Cluster.

The reason to document it here is twofold:

  1. Information often it scattered 'all over the place', but never 100% applicable to the setup I have.
  2. To remember 'what the fuck' did I do some months ago.
  3. Writing it for 'a public' forces me to think it all through again and make sure it's correct.

It's written 'first to scratch my own itch' but hopefully it benefits others too, or even better, that others improve upon my implementations. Feel free to comment or share improvements and insights!

Create Erasure Coded CephFS Pools

Part of collection: Hyper-converged Homelab with Proxmox

How to create a Erasure Coded Pool in Ceph and use 'directory pinning' to connect it to the CephFS filesystem.

To use a Erasure Coded Pool with CephFS, a directory inside the CephFS filesystem needs to be connected to a Erasure Coded Pool, this is called 'directory pinning'.


Mount Volumes into Proxmox VMs with Virtio-fs

Part of collection: Hyper-converged Homelab with Proxmox

Virtio-fs is a shared file system that lets virtual machines access a directory tree on the host. Unlike existing approaches, it is designed to offer local file system semantics and performance. The new virtiofsd-rs Rust daemon Proxmox 8 uses, is receiving the most attention for new feature development.

Performance is very good (while testing, almost the same as on the Proxmox host)

VM Migration is not possible yet, but it's being worked on!

Docker Swarm in LXC Containers

Part of collection: Hyper-converged Homelab with Proxmox

After struggling for some days, and since I really needed this to work (ignoring the it can't be done vibe everywhere), I managed to get Docker to work reliable in privileged Debian 12 LXC Containers on Proxmox 8

(Unfortunately, I couldn't get anything to work in unprivileged LXC Containers)

There are NO modifications required on the Proxmox host or the /etc/pve/lxc/xxx.conf file; everything is done on the Docker Swarm host. So the only obvious candidate who could break this setup, are future Docker Engine updates!