Backups that don't make me nervous

I run Proxmox Backup Server (PBS) on a fourth node in the homelab. The PBS node is the only one that’s not part of the Proxmox cluster. It just sits there, takes backups, and verifies them. It’s the most important node and the least interesting one.

Here’s the setup.

The PBS host

Mini-ITX motherboard
Intel N100 (low power, 6W TDP, plenty for PBS)
32 GB RAM
2x 8 TB HDDs in ZFS mirror
One 1GbE NIC

The whole node draws about 20W at idle. It runs 24/7. The electricity cost is about $20/year. The PBS software is free.

The backup jobs

I have a backup job for each VM. The job runs nightly. The schedule is staggered so all the VMs don’t try to back up at the same time.

VMs: nightly, keep 7 daily, 4 weekly, 12 monthly
PBS datastore: main on the ZFS mirror
Verification mode: verify after each backup job (not the default)

The verify step is the part that matters. By default, PBS just writes the backup. To verify, PBS reads the backup back from disk and checksums it against what was originally written. This catches bit rot on the backup disk. It doubles the I/O and the time, but backups are the one place where you want the extra work.

The retention that actually works

The retention numbers I picked aren’t random:

7 daily = one week of recent backups for the “I deleted the wrong file yesterday” case
4 weekly = one month of backups for the “I broke something and only noticed today” case
12 monthly = one year of backups for the “we got hit with ransomware six months ago” case (this is the one people skip and then regret)

The actual storage cost is manageable because PBS uses deduplication and compression. The full backup of all my VMs is about 2 TB. The deduplicated size on the PBS side is about 1.4 TB. The compression brings it down to about 1 TB on disk. The 8 TB mirror has plenty of room for years of retention at this rate.

The verification schedule

The nightly backup job runs verify after each backup. That catches write-time errors. For read-time errors (bit rot that happens between backup runs), I have a weekly job that runs prune and garbage-collect plus a verify of all snapshots older than 30 days. The weekly verify is what catches silent corruption.

If the weekly verify reports any snapshot as corrupt, I get an email. The email is sent by a tiny cron job that runs proxmox-backup-manager and parses the output. The script is about 30 lines. It’s on the PBS host.

The restore test

Once a quarter, I do a full restore drill. I pick a random VM, restore it to a new VMID on the cluster, boot it, verify the data is there, then delete the test VM. The drill takes about 20 minutes. It catches:

Backups that are running but not actually capturing the data (I’ve had this happen twice, both times because of a misconfigured snapshot mode)
Backups that are complete but the restore path is broken (had this once, NFS permissions issue)
VMs that depend on something outside the backup (database connection strings, API keys) and don’t work after restore

The drill is annoying. I do it quarterly because twice a year I find something wrong with the backups. The cost of finding it during a real restore is much higher than the cost of finding it during the drill.

The thing I don’t back up

I don’t back up the Proxmox hosts themselves. If a host dies, I reinstall Proxmox on the same hardware, add it to the cluster, and the VMs come back from PBS. The host OS is not interesting. The VMs are.

What I’d add if I had more budget

A second PBS host in a different physical location. The PBS host is in the same closet as everything else. A fire would take out the cluster and the backups. A second PBS host at a friend’s house, connected over a WireGuard tunnel, syncing the most recent week of backups, would protect against the “the house is gone” case. I don’t have that yet. It’s the next thing on the list.