septicnerd.dev

Setting up GPU passthrough for local LLM serving

GPU passthrough on Proxmox is the difference between “I have a server” and “I have a server that can run local models.” The setup is not hard. The setup is also not documented in one place. Here’s what I did.

The hardware

  • Host: one of the Proxmox nodes (the one with the GPU)
  • GPU: RTX 3090, 24 GB VRAM
  • CPU: Ryzen 5800X (no integrated graphics, which is actually what you want)
  • Motherboard: ASRock Rack X570D4U-2T (has the right BIOS options)

The 3090 is a great local LLM card for the money. 24 GB is enough for quantized 70B models, plenty for 13B, overkill for 7B. The 3090 is also old enough that the prices have come down. The 4090 is faster but the 3090 is the budget pick.

The BIOS settings

  • SVM (Secure Virtual Machine) enabled
  • IOMMU enabled
  • ACS Enable (this is the part that matters for multi-GPU systems; the 3090 is the only GPU so I didn’t need it but I enabled it anyway)
  • Above 4G Decoding enabled
  • Resizable BAR: doesn’t matter for passthrough, leave default

Save and reboot. The host now has IOMMU groups. Verify with:

find /sys/kernel/iommu_groups/ -type l | sort -V

You should see a list of IOMMU groups, each with the PCI devices in that group. The GPU and its audio function should be in their own group. If they’re in a group with other devices, you have an IOMMU grouping problem and ACS override is the workaround. I didn’t need it.

The Proxmox config

Edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Update grub and reboot. The iommu=pt (pass-through) is the part that lets the CPU skip the translation layer for assigned devices. It’s faster but less safe. For a homelab, faster is fine.

Edit /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:2204,10de:1aef disable_vga=1

The two PCI IDs are the GPU and the audio function. Get them with lspci -nn | grep -i nvidia.

Update initramfs and reboot. The GPU is now bound to the vfio-pci driver and ready to be passed through.

The VM setup

  • Create a VM with the right CPU type (host is fine for a homelab)
  • Allocate all the RAM and all the cores you can spare
  • In the Hardware tab, add a PCI Device and pick the GPU
  • Check “All Functions” to grab the audio function too
  • Check “Primary GPU” if you want to see the boot screen
  • In the Options tab, set the Machine type to q35 (newer, better for passthrough)

Start the VM. Install Ubuntu 24.04 or whatever you like. Install the NVIDIA driver. Reboot. The VM should have the GPU.

The software

For local LLM serving I use ollama. It’s a single binary, no configuration, pulls the model on first request. The VM has ollama running on port 11434. I expose that port to the host network. From the host, I can hit http://ollama-01.lan:11434 and the model answers.

The whole thing took about two hours, including reading the docs and rebooting four times. The longest part was waiting for the NVIDIA driver to compile.

What doesn’t work

  • Display output. The VM sees the GPU but I’m not using it for a display. Headless setup.
  • Suspend and resume. Passing through a GPU breaks VM suspend. The VM has to be fully shut down, not suspended. I scripted around it.
  • Migration. A VM with a passed-through GPU cannot be live-migrated to another node. The GPU is bound to one host. The VM stays on that host until I shut it down.

The part I got wrong the first time

I enabled the wrong IOMMU setting. There’s amd_iommu=on and there’s amd_iommu=force_isolation. They’re not the same. The first is for AMD’s IOMMU. The second is for an obscure feature that does something else. I had the second one in my config for six months before I noticed. The first one is what you want for passthrough.

If your VM boots but the GPU shows as “unknown device” in the guest, the IOMMU is not on. Check the kernel command line. Check dmesg | grep -i iommu. It will tell you whether IOMMU is on and which mode it’s in.

← back to all posts