General Score: 8/10
Intel Xeon Broadwells E5 v1 or v2s.
- These are pretty slow, but they work great for servers where single-threaded performance is not important. Their core counts are terrible (8 cores or 10 cores), but you can find these for around $20 so.
Mine is configured with a pair of E5-2670 v1 for 8x2 cores = 16 cores. It works great for the most part.
A lot of DDR3.
- This server can hold a crap ton of RAM. I think the max amount is ~768GB, but I may be wrong.
Mine is configured with 192GB of ECC RAM (a lot). I've never ran out of RAM, and DDR3 is extremely cheap these days.
Very picky. 4x single slot or 2x double slot.
- This server has PCIe lanes orientated on risers. Depending on the risers, you will either get 4 x8 lanes, 2 x16 lanes, or 1x 16 and 2x8. I don't remember riser numbers.
- Delivering power to GPUs requires specific cables. The standard case has an input of EPS 4+4 and a proprietary PSU connector. Do not use CPU PSU cables or your GPU cables. Use the Dell R730 cable for datacenter GPUs and the official R720 GPU cable for consumer GPUs (6+2) cable.
- The max power deliver for each GPU is 75w (PCIe) + 225w (connector) for a total of 300w. Don't use GPUs with higher TDP. Mine is configured with riser 2 + 3, giving me 2 x8 slots and 1 x16 slot. I have 3 spare x8 slots from the third riser, but I haven't found a good use case yet.
Aside from that, I have 2x NVIDIA P100s in my server, each with 12GB VRAM. I use the official R730 cables after having issues with a Corsair PSU CPU cable. The chassis to fit these GPUs is very tight. I dropped one (lol) and dented its plate; it was very difficult to get into the server.
A crap lot of HDDs.
- This server has a front chasis allowing 16x 2.5 disks or 8x 3.5 disks. This is not changeable and depends on the chassis model you got.
Mine is equipped with 8x 3TB 3.5 HDD disks @10K RPM for a total of 24TB disk. RAID-6 bumps my storage down to 16TB, which is still plenty.
The boot time is extremely long, but I'm too broke to upgrade to SSDs. A single one costs ~100, so 8x would cost $800. Too broke, moving on.
4x 1-10GbE ethernet + 1GbE iDRAC.
- This server has 4 ports for ethernet and a dedicated port for iDRAC. These ports can be upgraded by swapping PCIe cards if you need higher speeds. The iDRAC port can only be used with iDRAC enterprise.
Mine is equiped with 4x 1GbE, as I've never found the need for 10GbE (~$100). I do use iDRAC and have enterprise, but I'll talk about iDRAC later, so don't worry if you don't know what it is.
2x 750w or 2x 1100w
- This server can handle power supplies of 750w or 1100w. 1100w is for capping the GPUs (2x 300w) + a whole lot of PCIe devices + very high TDP cpus.
Mine is equiped with 2x 750w PSUs running in redundancy mode. Despite the wattage seeming terrible for 2 GPUs, it is faring pretty well.
Enough with specs, let's get to the cost!
- Chassis + components refurbished from Amazon: $300
- 2x P100s: $400
- 2x R730 cables: $10
- Eletricity/Month: ~256w/hr ~= $300/yr
Cloud (Linode) Compute:
Yeah... Expensive. Our server is providing some pretty sweet cost savings + worryfree billing. But we haven't factored in maintenance time! On a typical year, I probably spend around 14 days troubleshooting this server, and spend around 5 hrs on each day. That places it at 70hrs/year. At minimum wage ($15, don't hire me as system admin lol), this gets around $1050/yr.
It still nets about less than cheap cloud so, pretty nice. And that cloud machine doesn't even have GPUs (or the amount of CPU, RAM, whatever)!
I think the server is pretty good for the price. I give it a solid 8/10, would recommend if broke. Here are some cons:
- Very power hungry
- Very loud (to cool GPUs)
- Slow CPUs in terms of singlethreaded (works fine on Minecraft remote though)
- NGINX & HAProxy (LB)
- PostgreSQL (DB)
- Grafana (Monitoring)
- Pi-hole (Router, DNS)
- Nextcloud (Google Drive)
- Proxmox (Virtualization)
- Rocky, RHEL, Debian, and ArchLinux (OS)
- JuypterLab (Data Science work)
- Tailscale (VPN, exit nodes)
- Gitlab & Gitea (Source Code Management)
- Gitlab/Github runner (CI/CD)
- NoMachine (Remote Software)
- Coder (Online Coding Workspace Manager)
- A lot of internal apps.
In the end, I think cloud is better for professional/company work. If your app needs 95 SLO, please do not use a homelab like mine. It's way easier to just add cloud, and save developers from having to navigate through painful internal stacks.
But I think homelabs are better than cloud for learning and individuals. IMO it's extremely easy to transfer homelab skills to cloud as cloud is simply just an abstraction to a homelab. Google Load Balancer? Try setting up your own NGINX. Database? Try running a PostgreSQL + PGAdmin. It's extremely easy to go from homelab to cloud.
The other thing is that costs is such a worrysome thing. I hate panicking over cloud bills for a home hobby project. It's annoying. With the homelab, the monthly cost is pretty dirt cheap and can never go up. Pretty cool.
Google searches.
99% of the time you will run into some obselete error. It's like daily driving ArchLinux, things go south extremely easily when playing with Wine or obsecure software. The same goes with your server. You won't find every solution on this post. But you know, someone somewhere in the world had the same problem as you. They made a post on reddit or somewhere, and people came and helped them and they fixed it.
Your job as an system administrator isn't to check if everything is working; your job is to fix obselete problems that no one knows how to. And researching what kind of solutions are the key; TLDR, get EXTREMELY good at googling and debugging.
- Dell Site
- 99% of this doc can be attained with google searches onto forums. This place is a nice aggregation of that.