septicnerd.dev

Homelab tour: my Proxmox cluster in a closet

I run a three-node Proxmox cluster out of a closet in my office. The closet is not a server closet. It’s the kind of closet where you keep coats and Christmas decorations. There is no rack. There is no cooling. There is a lot of dust.

Here’s the setup.

The hardware

Three nodes, all the same:

  • ASRock Rack X570D4U-2T motherboard
  • AMD Ryzen 7 5800X
  • 64 GB ECC DDR4
  • 2x 1 TB NVMe (mirror, ZFS)
  • 1x 4 TB SATA SSD (for VMs that need more space)
  • Intel X550-T2 dual-port 10GbE SFP+
  • 250W SFX PSU

The motherboard has IPMI. The IPMI is what makes the closet livable. I can reboot any node from my laptop without going near the closet. I can see the POST screen. I can mount a virtual CD for reinstalls. The IPMI alone is worth the price of the board.

The network

  • One 10GbE switch (MikroTik CRS305)
  • One 1GbE switch for the management network (a leftover Netgear)
  • The two switches are connected by a single 10GbE DAC cable
  • Each node has both 10GbE and 1GbE connected
  • 1GbE is the management network (Proxmox web UI, IPMI)
  • 10GbE is the VM traffic and the storage replication
  • The cluster uses a separate VLAN for the Ceph replication traffic

I don’t have a separate storage node. Storage is local to each node, replicated via Ceph. Ceph on three nodes with one replica is fine. With two replicas it’s slow. With three replicas I run out of disk. One replica is the budget choice.

What’s running

  • pihole-01 (DNS sinkhole, plus Unbound for recursive resolution)
  • nas-01 (TrueNAS Scale, exports NFS and SMB to the LAN)
  • homeassistant-01 (the home automation, the lights, the thermostat)
  • gitea-01 (self-hosted Git, mostly for the homelab repo itself)
  • paperless-01 (ingesting all the paper mail and receipts)
  • jellyfin-01 (media server, mostly for the kids’ stuff)
  • ollama-01 (local LLM serving, RTX 3090 passed through)
  • adguard-01 (fallback if pihole dies, with synced config)

I have 24 other VMs defined but not running. They’re templates. When I need a new VM, I clone from a template, change the hostname, and start it. Cloning a template takes about 12 seconds.

The closet itself

The closet has:

  • A 12U open-frame rack (StarTech, the cheap one)
  • The three nodes mounted on the rack shelf
  • The two switches on a 1U shelf above the nodes
  • A small APC UPS (1500VA) powering everything
  • A cable management arm that I bought and never installed

The temperature in the closet runs about 78F ambient. The nodes report CPU temps in the low 60s C under load. The fans ramp up when the LLM is running. The whole closet gets noticeably louder for the 30-60 seconds the GPU is hot.

What I’d do differently

  • I’d buy two of the 10GbE switches instead of one. The single switch is a SPOF.
  • I’d mount the nodes with rails. They sit on a shelf and slide when I bump them.
  • I’d put the IPMI on a separate VLAN. Right now it’s on the same network as my laptop. Theoretically fine. Practically a thing I should fix.
  • I’d skip the TrueNAS VM and just use a ZFS dataset on one of the nodes. The VM is one more thing to maintain.

Total cost when I built it: about $4,200 for all three nodes and the switch. Total cost now: I’d build the same thing again, almost. The homelab pays for itself in the projects I get to run on it.

← back to all posts