Loading

Back to Home

Homelab Rebuild Part 1: Upgrading Proxmox VE 8 to 9

Technology Homelab Proxmox
| October 04, 2026 | 8 min to read

Intro

My homelab runs a Kubernetes cluster on three Proxmox hosts. Before rebuilding that cluster, the hosts themselves needed to move from Proxmox VE 8 (Debian 12 bookworm) to Proxmox VE 9 (Debian 13 trixie). This series is the story of that whole rebuild, from the hypervisors up to a cluster with secrets management, TLS, DNS, CI, a database and a public entry point. This first part is the hypervisor upgrade.

TL;DR: Upgrade one node at a time with /etc backed up, a simulated dist-upgrade first, and the real one run under systemd-run so an SSH drop can’t kill it. Then expect cleanup: PVE 9 adds the enterprise repo back, an expired Grafana apt key blocks updates, inter-node root SSH breaks, and it’s a good moment to notice what your monitoring doesn’t cover.

The Situation

HostCPURAMStorageGuests during the upgrade
pve-01AMD, 8 cores30 GBNVMe local-lvm + SATA ssdone VM
pve-02AMD, 8 cores30 GBNVMe local-lvm + SATA ssdnone
pve-03Intel, 8 cores7.6 GB68 GB local-lvm thin poolone VM
  • Before: pve-manager/8.4.19, kernel 6.8.12-36-pve
  • After: pve-manager/9.2.21, kernel 7.0.14-23-pve
  • No Ceph, only local LVM-thin storage. Quorum stayed 3/3 the whole time.
  • Upgrade order: pve-02, pve-03, then pve-01 (the busiest node last). Each dist-upgrade took about 7 to 8 minutes, around 610 to 640 packages per node.

Setup

Preparation on all nodes

pve8to9 --full is the official checker. I ran it, fixed what it reported (Issue 2), brought every node to the latest 8.4, and ran it again until it showed 0 failures:

pve8to9 --full
cp -a /etc/apt/sources.list /root/sources.list.bak-$(date +%F)
apt-get update && apt-get -y -o Dpkg::Options::=--force-confold dist-upgrade
pve8to9 --full
# = SUMMARY = ... FAILURES: 0

The per-node playbook

  1. Back up /etc:

    tar czf /root/etc-backup-pre-pve9-$(date +%F).tgz /etc

  2. Switch the repositories to trixie. One sed covered Debian, pve-no-subscription, the Ceph no-subscription repo and Tailscale:

    sed -i 's/bookworm/trixie/g' /etc/apt/sources.list /etc/apt/sources.list.d/*.list

  3. Simulate first, and look for removals of anything critical:

    apt-get update
apt-get -s dist-upgrade | grep '^Remv' | grep -E 'pve|proxmox|qemu-server|corosync|lvm2|grub'
# (nothing)

  4. Stop the guests on that node (qm shutdown <vmid> --timeout 180).

  5. Run the upgrade as a transient systemd unit, so a dropped SSH session can’t interrupt dpkg halfway. --force-confold keeps my existing configs; the package versions are saved next to them as *.dpkg-dist:

    systemd-run --unit=pve9-upgrade \
  --setenv=DEBIAN_FRONTEND=noninteractive --setenv=APT_LISTCHANGES_FRONTEND=none \
  bash -c 'apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold \
    dist-upgrade > /root/pve9-upgrade.log 2>&1; echo EXIT=$? >> /root/pve9-upgrade.log'

    Then poll the log until EXIT= appears.

  6. Check before rebooting: pveversion, dpkg --audit, the new kernel in /boot, and pve8to9. Two warnings are expected at this point:

    WARN: a suitable kernel (proxmox-kernel-7.0) is installed, but an unsuitable (6.8.12-36-pve) is booted, missing reboot?!
WARN: No (active) time synchronisation daemon (NTP) detected

    The NTP one was chrony failing against the old systemd (Could not send notification to $NOTIFY_SOCKET). The reboot fixed both.

  7. Reboot, and wait for quorum from another node:

    systemd-run --on-active=3 systemctl reboot
# from a peer:
until pvecm status | grep -q 'Total votes:      3'; do sleep 10; done

  8. Post-checks: pve-cluster, corosync, pvedaemon, pveproxy and chrony active, no failed units, pve8to9 at 0 warnings and 0 failures. Then start the guests again.

Configs kept by --force-confold (with the new defaults saved as *.dpkg-dist): /etc/issue and /etc/lvm/lvm.conf everywhere, /etc/systemd/logind.conf on pve-03 and /etc/lxc/default.conf on pve-01. Worth a diff later, not urgent.

Issue 1: A VM Froze on pve-03 Before I Started

Symptom

The VM on pve-03 was paused with qmpstatus: io-error. The kernel log said:

device-mapper: thin: 252:4: switching pool to out-of-data-space (error IO) mode

Cause

The 64 GB local-lvm thin pool held a VM disk with a 100 GB virtual size, and discard was off, so blocks freed inside the guest were never returned to the pool. It filled to 100%.

Fix

Give the pool a little room, resume the VM, then enable discard and trim:

lvextend -L +4G pve/data
qm resume 400
qm set 400 --scsi0 local-lvm:vm-400-disk-0,discard=on,ssd=1,size=100G
qm reboot 400
qm guest cmd 400 fstrim
# pool usage dropped to about 14%

Thin pools overcommit by design. Without discard, they only ever grow.

Issue 2: pve8to9 Failures and Warnings

Symptom

FAIL: systemd-boot meta-package installed. This will cause problems on upgrades of other boot-related packages. Remove 'systemd-boot'
WARN: matching CPU microcode package 'amd64-microcode' could not be found!
WARN: storage 'ssd' enabled but not active!

Cause

  • These nodes boot with GRUB and shim, not systemd-boot. The meta-package was installed but unused.
  • Microcode packages live in Debian’s non-free-firmware component, which my sources didn’t include.
  • The ssd storage only exists on pve-01 and pve-02, but was enabled for all nodes.

Fix

apt-get -y remove systemd-boot          # keeps systemd-boot-efi
sed -i -E '/debian.org/ { /non-free-firmware/! s/(main contrib)$/\1 non-free-firmware/ }' /etc/apt/sources.list
apt-get update && apt-get -y install amd64-microcode   # intel-microcode on pve-03
pvesm set ssd --nodes pve-01,pve-02

Issue 3: The Enterprise Repo Came Back

Symptom

After the upgrade, the repository list showed the enterprise repo enabled on every node, which fails without a subscription.

Cause

The PVE 9 packages add /etc/apt/sources.list.d/pve-enterprise.sources (deb822 format, Suites: trixie, Components: pve-enterprise), even if the old .list version had been removed.

Fix

Move it out of the way (keeping a copy) and confirm through the API:

mv /etc/apt/sources.list.d/pve-enterprise.sources /root/pve-enterprise.sources.bak-$(date +%F)
pvesh get /nodes/$(hostname)/apt/repositories

The pve-no-subscription and Ceph no-subscription repos stay. A future upgrade may bring the file back again, so it’s worth checking after every major update.

Issue 4: The Grafana Apt Key Had Expired

Symptom

On bookworm:

GPG error: https://apt.grafana.com stable InRelease: The following signatures were invalid: EXPKEYSIG 963FA27710458545 Grafana Labs <[email protected]>

On trixie, apt’s new verifier (sqv) reports the same thing more verbosely:

Signing key on B53AE77BADB630A683046005963FA27710458545 is bad: The primary key is not live because: Expired on 2025-08-23T16:33:45Z

Cause

Grafana extended the expiry of the same key. The fingerprint didn’t change, but my keyring still held the old copy with the old expiry date.

Fix

Download the key again:

cp -a /etc/apt/keyrings/grafana.gpg /root/grafana.gpg.bak-$(date +%F)
curl -fsSL https://apt.grafana.com/gpg.key | gpg --dearmor > /tmp/grafana.gpg
install -m 644 /tmp/grafana.gpg /etc/apt/keyrings/grafana.gpg

Same fingerprint, now valid until 2027-08-22. A note to refresh it before then went into my notes.

Issue 5: Root SSH Between Nodes Broke

Symptom

Host key verification failed.

After the last node was upgraded, 4 of the 6 node-to-node directions failed. Proxmox’s own features (migration, console) kept working.

Cause

Plain ssh relied on /etc/ssh/ssh_known_hosts, the old symlink into /etc/pve/priv/known_hosts. After the upgrade it was gone on pve-02 and pve-03, and on pve-01 it only held pve-01’s own key. Proxmox’s own SSH calls pin host keys with HostKeyAlias and UserKnownHostsFile=/etc/pve/nodes/<node>/ssh_known_hosts, which is why they were unaffected. The peer hostnames also resolve to Tailscale IPs, not the cluster ring addresses.

Fix

Fill root’s known_hosts from the keys Proxmox already distributes, rather than trusting whatever ssh-keyscan returns on first contact:

cp -a /root/.ssh/known_hosts /root/.ssh/known_hosts.bak-$(date +%F)
for peer in pve-01 pve-02 pve-03; do
  # names: hostname, its corosync ring0 address and what it resolves to (Tailscale)
  names="$peer,$(awk -v n="$peer" '$1=="name:"&&$2==n{f=1} f&&$1=="ring0_addr:"{print $2; exit}' /etc/pve/corosync.conf),$(getent hosts $peer | awk '{print $1}')"
  awk -v n="$names" '{print n, $(NF-1), $NF}' /etc/pve/nodes/$peer/ssh_known_hosts >> /root/.ssh/known_hosts
done

Issue 6: Monitoring Only Covered One Node

Symptom

Grafana Alloy (1.10.2) was installed only on pve-01, so the other two hypervisors sent no metrics or logs to Grafana Cloud at all.

Fix, and the gaps it exposed

I installed the same Alloy version on pve-02 and pve-03 with an identical config. It ships node-exporter metrics, the PVE API’s Prometheus endpoint (/api2/prometheus/metrics) and the journal plus a few log files, and drops any log line containing password. I tightened the config file permissions to 640 root:alloy. Reviewing the config across three nodes surfaced the real problems:

  • The PVE API token had no permissions of its own. It was created with privilege separation, but only the user had the PVEAuditor role. The token needs its own ACL (or --privsep 0).
  • host = env("HOSTNAME") is empty under systemd, so every node would report the same host label. Use constants.hostname instead.
  • Cluster-wide PVE metrics would be scraped three times, once per node.
  • A token value was printed in plain text while debugging, so it has to be rotated before going live.

The service stays disabled on all three nodes until the Grafana Cloud side is set up properly. Shipping metrics with these gaps would have looked like monitoring without being it.

Lessons

  1. Run long upgrades under systemd-run. An SSH drop in the middle of dpkg is how a hypervisor ends up half-upgraded.
  2. Simulate, then grep for removals. A dist-upgrade that removes corosync or pve-manager should never be the first surprise.
  3. Re-check repositories after every major upgrade. Packages can add sources back.
  4. An expired key is often the same key, extended. Download it again before hunting for a new one.
  5. Upgrades are a good time to audit monitoring. “Installed” is not “covered”, and three identical configs show what one config hides.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9 (this post)
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers

References