Shortly after starting my career in IT, my team started to embrace the DevOps mentality and it's been pretty hard to look back. As a developer in today's world, I feel lucky and empowered to have access to tools like Ansible & Terraform, so a big part of this is fueled by a desire to strengthen my DevOps skills. Personally, I have found that wearing many hats is key to becoming a more well-rounded programmer as well. And that means learning about all the pieces involved, not just the application layers. So the days of working in a black box are over for me!
Over time, I also became much more educated on the issues of data privacy and how we blindly trust Tech Giants to not misuse our information. It worries me that often times we are not even aware there is any manipulation happening. I think that fear, combined with the current state of cybersecurity is what finally hit it home for me. I asked myself, "How far could I actually go with owning my own data?" I was familiar with the idea of self-hosting but thought that it was probably a niche thing now, ever since the big push to cloud computing. When I stumbled upon Home-Labbing and the extremely supportive community surrounding it, I was pleasantly surprised and relieved that my assumptions were wrong.
So that's when I decided I would build something that would mimick my work's on-prem datacenter, but at home, for personal use. I figured it would not only strengthen my skills, but could actually help liberate me and my data, while being fun in the process. I will admit, that I caught The FOSS Bug in the process, but now my eyes are open to a plethora of hidden gems that I was blind to before. The home lab is quite literally my oyster 😉.
Before setting out on a shopping spree, I needed to have an idea of what bare-metal operating system I would be running as this might affect the hardware compatibility. At work we were using VMWare's VSphere but thought the Web Interface was clunky and it felt wayy to proprietary for my personal needs. Proxmox Virtual Environment was an easy choice for me, being open-source and being based on Debian. I also love how the free (non-subscription) license is not crippled by any means (unlike VMWare's ESXI). Out of this came my only notable hardware contraint, which was to make sure things worked well with the linux kernel, like performance and drivers.
Aside from my hypervisor-of-choice, there were some phyical constraints I was keeping in mind as well. Due to being a renter, limited to small spaces and moving periodically, I wanted something that could be packed up and in the car in a matter of minutes, not hours or days. This meant no rack mounts and fancy cable management (for now), to name just a few sacrifices. So ideally this had to be portable, quiet, low heat, and power-efficient. What I ended up with, was a little over-budget but it's been worth every penny.
8 Core (16 thread) Xeon CPU, 64 GB ECC Memory, with 4 TB local NVMe Storage, and a GPU thrown in, primarily for plex transcoding, but figured it would be fun to dabble with AI/ML workloads 🎉
- 1 x Intel® NUC 9 Pro Kit - NUC9VXQNX (Quartz Canyon, Bare bones kit)
- Aluminum/plastic housing
- Built-in 500 Watt platinum rated power supply
- 2 x Front USB 3.1 (enabled in BIOS, for emergency peripheral usage)
- 1 x SDXC slot with UHS-II support (disabled in BIOS, not used)
- 1 x integrated Intel Wifi6 AX200 (WiFi/BT Module) PCI device
- Just a side note here. This is fun to keep enabled as you can pass it through to Virtual Machines and do some "friendly neighborhood WiFi pentesting" in your apartment complex ;)
- 1 x empty NVMe M.2 PCIe x4 slot (CPU)
- 1 x Intel® Optane H10 (1 TB M.2 2280 PCIe 3.0 x4 SSD)
- 1 x empty PCIe x16 (in x8 mode, since one of either M.2 x4 slot or PCIe x4 slot is used)
- 1 x Nvidia Quadro P2200 (5 GB GDDR5X PCIe 3.0 x16, with unlocked NVENC transcoding)
- 1 x empty PCIe x4 slot
- 1 x NVMe PCIe Adapter (M.2 2280 NVMe SSD to PCIe x4 Converter Card)
- 1 x Intel® 665p (1 TB M.2 2280 PCIe NVMe 3.0 x4 3D3, QLC SSD)
- 1 x NVMe PCIe Adapter (M.2 2280 NVMe SSD to PCIe x4 Converter Card)
- 1 x Intel® NUC 9 Pro Compute Element - NUC9VXQNB
- 1 x Intel® Xeon E-2286M Processor (16 logical cores @ 2.40 GHz)
- 1 x Intel® UHD Graphics P630
- 2 x empty DDR4 SDRAM SO-DIMM slots
- 2 x 32 GB ECC Memory (DDR4-2666 PC4-21300 ECC 2Rx8 SODIMM)
- 2 x empty NVMe M.2 PCIe x4 slots (PCH)
- 2 x Intel® 665p (1 TB M.2 2280 PCIe NVMe 3.0 x4 3D3, QLC SSD)
- 4 x Rear USB 3.1 (not currently used for anything)
- 2 x 1 gigabit Ethernet (i219-LM & i210-AT)
- One port is used for WAN side, other for dedicated management (super helpful if you fuck something up while configuring VLANs or the like 😅)
- 2 x Thunderbolt 3 (Can support 10 Gigabit NICs, backwards compatible with USB C)
- 2 x USB C to Ethernet Adapter (1 gigabit NICs)
- I'm using these cheap adapters for the time being that are configured in LAG for my LAN side of things, no issues whatsoever and frees up my other "real" ethernet ports
- 1 x HDMI 1.0a
- 1 x HDMI Display Emulator (Headless Ghost Dongle, 1920x1080 @ 60 Hz)
- Not necessary for typical usage but found it helps tremendously with weird remote OOB management edge cases. You can consider one of these required if you want to have a fully-feature-enabled Intel AMT with remote desktop capabilities. For example, if you plan on using Mesh Commander and wish to boot to BIOS remotely you need one of these in order to see the BIOS password prompt screen.
- 1 x HDMI Display Emulator (Headless Ghost Dongle, 1920x1080 @ 60 Hz)
- 1 x Intel® Xeon E-2286M Processor (16 logical cores @ 2.40 GHz)
This is actually my newest piece of equipment and picked this up for pretty cheap on Amazon recently in order to play around with and learn more about VLANs. This is a managed switch with Layer 3 features and Power-over-Ethernet.
- 1 x Cisco SG250-08HP
I happened to already own this router before I got into homelabbing, and so I just repurposed it by putting in in bridge mode after virtualizing PfSense as my WAN router/firewall. The wireless radio on this thing is awesome, but to my knowledge DDWRT has no WiFi 6 support yet, so configuring VLAN SSID's is close to impossible without writing super niche scripts for MerlinWRT. This obviously makes running an isolated guest networks on the same AP a little more challenging. I will probably move to something like Ubiquity's UniFi APs eventually since it has both proper VLAN and WiFi 6 support already.
- 1 x Asus RT-AX88U running AsusWRT-Merlin in Bridge Mode (Access Point Mode)
I was new to NAS at the time, and wanted something that just worked and was very small and portable. I also knew that this would be primarily used for storing relatively cold-warm data, nothing hot as I had plenty of very fast local storage in the NUC. I would probably build by own NAS system moving forward by using something like TrueNAS. Synology products are generally overpriced in my opinion, especially for those who don't care for the Web interface that much.
- 1 x Synology DS920+ For VM Backups and Network File Services
- 1 x 4 GB Memory module (added to the included 4 GB to make 8 GB total RAM)
- 4 x 8 TB Seagate IronWolf internal HDD in RAID 5 with
btrfsfilesystem asvolume1(20.9 TB Usable space) - 2 x 1 TB Samsung EVO 970 Plus SSD in RAID 1 as Read/Write Cache for
volume1
For the unfortunate scenario where coaxial is the only option, I picked up one of these and made sure it supports DOCSIS 3.1 for the superior speeds. For fiber connections, some ISPs allow PPPoE to deliver your internet connection. I was able to set this up with Centurylink Fiber in Seattle as I had direct access to the fiber-to-ethernet box that was installed in my garage.
- 1 x Arris SURFBoard SB8200 Cable Modem
PVE Dashboard and VM's..
My local storage SSD's are wearing out quick, but thank goodness the disk striping seems to be working correctly across my zfs pool..
I run all my services on Docker Swarm. IMO Kuberentes is overkill for most home labs. You can save some resources using a lighter orchestration engine. Here is what I currently have running on my tools single-node swarm host..
Just giving you a taste of my internal network. I use Step-CA with Traefik to automatically issue/renew internal certs uaing ACME, so TLS is on everything. Both internally and externally. This is Organizr dashboard, soon to be replaced with my own :)
Coming soon!
I use a GitHub to not only store my code but to help organize my home-lab efforts, configurations, scripts and anything else related. I try to keep as much of it public as possible as my way of knowledge sharing.
Kiwi Labs on GitHub 