Intro
My homelab runs a Kubernetes cluster on three Proxmox hosts. Before I rebuilt that cluster, the hosts themselves had to move from Proxmox VE 8 (Debian 12 bookworm) to Proxmox VE 9 (Debian 13 trixie). This series is the story of that whole rebuild. It goes from the hypervisors up to a cluster with secrets management, TLS, DNS, CI, a database and a public entry point. This first part is the hypervisor upgrade.
TL;DR: Upgrade one node at a time. Back up
/etc, simulate thedist-upgradefirst, and run the real one undersystemd-runso an SSH drop cannot kill it. Then expect cleanup: PVE 9 adds the enterprise repo back, an expired Grafana apt key blocks updates, and inter-node root SSH breaks. It is also a good moment to notice what your monitoring does not cover.
The Situation
| Host | CPU | RAM | Storage | Guests during the upgrade |
|---|---|---|---|---|
pve-01 | AMD, 8 cores | 30 GB | NVMe local-lvm + SATA ssd | one VM |
pve-02 | AMD, 8 cores | 30 GB | NVMe local-lvm + SATA ssd | none |
pve-03 | Intel, 8 cores | 7.6 GB | 68 GB local-lvm thin pool | one VM |
- Before:
pve-manager/8.4.19, kernel6.8.12-36-pve - After:
pve-manager/9.2.21, kernel7.0.14-23-pve - No Ceph, only local LVM-thin storage. Quorum stayed 3/3 the whole time.
- Upgrade order: pve-02, pve-03, then pve-01 (the busiest node last). Each
dist-upgradetook about 7 to 8 minutes, around 610 to 640 packages per node.
Setup
Preparation on all nodes
pve8to9 --full is the official checker. I ran it and fixed what it reported
(Issue 2). Then I brought every node to the latest 8.4 and ran it again until it
showed 0 failures:
pve8to9 --full
cp -a /etc/apt/sources.list /root/sources.list.bak-$(date +%F)
apt-get update && apt-get -y -o Dpkg::Options::=--force-confold dist-upgrade
pve8to9 --full
# = SUMMARY = ... FAILURES: 0
The per-node playbook
Back up
/etc:tar czf /root/etc-backup-pre-pve9-$(date +%F).tgz /etcSwitch the repositories to trixie. One
sedcovered Debian,pve-no-subscription, the Ceph no-subscription repo and Tailscale:sed -i 's/bookworm/trixie/g' /etc/apt/sources.list /etc/apt/sources.list.d/*.listSimulate first, and look for removals of anything critical:
apt-get update apt-get -s dist-upgrade | grep '^Remv' | grep -E 'pve|proxmox|qemu-server|corosync|lvm2|grub' # (nothing)Stop the guests on that node (
qm shutdown <vmid> --timeout 180).Run the upgrade as a transient systemd unit, so a dropped SSH session cannot interrupt
dpkghalfway. A transient unit is a one-off service that systemd runs outside your shell.--force-confoldkeeps my existing configs. The package versions go next to them as*.dpkg-dist:systemd-run --unit=pve9-upgrade \ --setenv=DEBIAN_FRONTEND=noninteractive --setenv=APT_LISTCHANGES_FRONTEND=none \ bash -c 'apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold \ dist-upgrade > /root/pve9-upgrade.log 2>&1; echo EXIT=$? >> /root/pve9-upgrade.log'Then poll the log until
EXIT=appears.Check before rebooting:
pveversion,dpkg --audit, the new kernel in/boot, andpve8to9. Two warnings are expected at this point:WARN: a suitable kernel (proxmox-kernel-7.0) is installed, but an unsuitable (6.8.12-36-pve) is booted, missing reboot?! WARN: No (active) time synchronisation daemon (NTP) detectedThe NTP one was
chronyfailing against the old systemd (Could not send notification to $NOTIFY_SOCKET). The reboot fixed both.Reboot, and wait for quorum from another node:
systemd-run --on-active=3 systemctl reboot # from a peer: until pvecm status | grep -q 'Total votes: 3'; do sleep 10; donePost-checks:
pve-cluster,corosync,pvedaemon,pveproxyandchronyactive, no failed units,pve8to9at 0 warnings and 0 failures. Then start the guests again.
--force-confold kept these configs and saved the new defaults as
*.dpkg-dist: /etc/issue and /etc/lvm/lvm.conf everywhere,
/etc/systemd/logind.conf on pve-03 and /etc/lxc/default.conf on pve-01.
They need a diff later, but it is not urgent.
Issue 1: A VM Froze on pve-03 Before I Started
Symptom
The VM on pve-03 was paused with qmpstatus: io-error. The kernel log said:
device-mapper: thin: 252:4: switching pool to out-of-data-space (error IO) mode
Cause
The 64 GB local-lvm thin pool held a VM disk with a 100 GB virtual size.
discard was off, so blocks that the guest freed never went back to the
pool. It filled to 100%.
Fix
Give the pool a little room, resume the VM, then enable discard and trim:
lvextend -L +4G pve/data
qm resume 400
qm set 400 --scsi0 local-lvm:vm-400-disk-0,discard=on,ssd=1,size=100G
qm reboot 400
qm guest cmd 400 fstrim
# pool usage dropped to about 14%
Thin pools overcommit by design: they promise more space than they have. Without
discard, they only ever grow.
Issue 2: pve8to9 Failures and Warnings
Symptom
FAIL: systemd-boot meta-package installed. This will cause problems on upgrades of other boot-related packages. Remove 'systemd-boot'
WARN: matching CPU microcode package 'amd64-microcode' could not be found!
WARN: storage 'ssd' enabled but not active!
Cause
- These nodes boot with GRUB and shim, not systemd-boot. The meta-package was installed but unused.
- Microcode packages live in the Debian
non-free-firmwarecomponent, and my sources did not include it. - The
ssdstorage only exists on pve-01 and pve-02, but it was enabled for all nodes.
Fix
apt-get -y remove systemd-boot # keeps systemd-boot-efi
sed -i -E '/debian.org/ { /non-free-firmware/! s/(main contrib)$/\1 non-free-firmware/ }' /etc/apt/sources.list
apt-get update && apt-get -y install amd64-microcode # intel-microcode on pve-03
pvesm set ssd --nodes pve-01,pve-02
Issue 3: The Enterprise Repo Came Back
Symptom
After the upgrade, the repository list showed the enterprise repo enabled on every node. That repo fails without a subscription.
Cause
The PVE 9 packages add /etc/apt/sources.list.d/pve-enterprise.sources
(deb822 format, Suites: trixie, Components: pve-enterprise). They add it even
when I removed the old .list version before.
Fix
Move it out of the way, keep a copy, and confirm through the API:
mv /etc/apt/sources.list.d/pve-enterprise.sources /root/pve-enterprise.sources.bak-$(date +%F)
pvesh get /nodes/$(hostname)/apt/repositories
The pve-no-subscription and Ceph no-subscription repos stay. A future upgrade
can bring the file back again, so check for it after every major update.
Issue 4: The Grafana Apt Key Had Expired
Symptom
On bookworm:
GPG error: https://apt.grafana.com stable InRelease: The following signatures were invalid: EXPKEYSIG 963FA27710458545 Grafana Labs <engineering@grafana.com>
On trixie, the new apt verifier (sqv) reports the same thing with more words:
Signing key on B53AE77BADB630A683046005963FA27710458545 is bad: The primary key is not live because: Expired on 2025-08-23T16:33:45Z
Cause
Grafana extended the expiry of the same key. The fingerprint did not change, but my keyring still held the old copy with the old expiry date.
Fix
Download the key again:
cp -a /etc/apt/keyrings/grafana.gpg /root/grafana.gpg.bak-$(date +%F)
curl -fsSL https://apt.grafana.com/gpg.key | gpg --dearmor > /tmp/grafana.gpg
install -m 644 /tmp/grafana.gpg /etc/apt/keyrings/grafana.gpg
Same fingerprint, now valid until 2027-08-22. I added a note to refresh it before then.
Issue 5: Root SSH Between Nodes Broke
Symptom
Host key verification failed.
After I upgraded the last node, 4 of the 6 node-to-node directions failed. The built-in Proxmox features (migration, console) kept working.
Cause
Plain ssh relied on /etc/ssh/ssh_known_hosts, the old symlink into
/etc/pve/priv/known_hosts. After the upgrade it was gone on pve-02 and
pve-03. On pve-01 it only held the key of pve-01 itself. The SSH calls that
Proxmox makes pin host keys with HostKeyAlias and
UserKnownHostsFile=/etc/pve/nodes/<node>/ssh_known_hosts. That is why they
still worked. The peer hostnames also resolve to Tailscale IPs, not the
cluster ring addresses.
Fix
Fill the known_hosts of root from the keys that Proxmox already distributes.
Do not trust whatever ssh-keyscan returns on first contact:
cp -a /root/.ssh/known_hosts /root/.ssh/known_hosts.bak-$(date +%F)
for peer in pve-01 pve-02 pve-03; do
# names: hostname, its corosync ring0 address and what it resolves to (Tailscale)
names="$peer,$(awk -v n="$peer" '$1=="name:"&&$2==n{f=1} f&&$1=="ring0_addr:"{print $2; exit}' /etc/pve/corosync.conf),$(getent hosts $peer | awk '{print $1}')"
awk -v n="$names" '{print n, $(NF-1), $NF}' /etc/pve/nodes/$peer/ssh_known_hosts >> /root/.ssh/known_hosts
done
Issue 6: Monitoring Only Covered One Node
Symptom
Grafana Alloy (1.10.2) was installed only on pve-01, so the other two
hypervisors sent no metrics or logs to Grafana Cloud at all.
Fix, and the gaps it exposed
I installed the same Alloy version on pve-02 and pve-03 with an identical
config. It ships node-exporter metrics, the Prometheus endpoint of the PVE API
(/api2/prometheus/metrics), the journal and a few log files. It drops
any log line that contains password. I tightened the config file permissions to
640 root:alloy. A review of the config across three nodes showed the real
problems:
- The PVE API token had no permissions of its own. I created it with
privilege separation, but only the user had the
PVEAuditorrole. The token needs its own ACL (or--privsep 0). host = env("HOSTNAME")is empty under systemd, so every node reports the same host label. Useconstants.hostnameinstead.- Cluster-wide PVE metrics get scraped three times, once per node.
- A token value showed in plain text while I was debugging, so I must rotate it before it goes live.
The service stays disabled on all three nodes until I set up the Grafana Cloud side properly. Metrics with these gaps look like monitoring, but they are not monitoring.
Lessons
- Run long upgrades under
systemd-run. An SSH drop in the middle ofdpkgis how a hypervisor ends up half-upgraded. - Simulate, then grep for removals. A dist-upgrade that removes
corosyncorpve-managermust never be a surprise. - Re-check repositories after every major upgrade. Packages can add sources back.
- An expired key is often the same key, extended. Download it again before hunting for a new one.
- Upgrades are a good time to audit monitoring. Alloy was installed, but it covered only one node. A comparison of three identical configs showed problems that one config hid.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9 (this post)
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed