Loading

Back to Home

Homelab Rebuild Part 10: When the Hypervisor Went Quiet

Technology Kubernetes Homelab Proxmox
| October 13, 2026 | 6 min to read

Intro

To cut electricity costs, I had just added a nightly power schedule: every VM shuts down at 22:00 and starts again at 06:00. The next morning everything came back, and half an hour later kubectl stopped answering.

What followed was two separate problems stacked on top of each other, plus one red herring that pointed at the wrong component.

TL;DR: One Proxmox host’s Realtek RTL8111/8168 NIC (r8169 driver) silently stopped passing traffic: no kernel error, link still UP. All three hosts had the same chip, so all three got EEE off and pcie_aspm=off. After recovery, a controller VM with 2 GiB and no swap thrashed its disk at 415 MB/s, and that one slow etcd member degraded the entire control plane.

The Situation

  • Three Proxmox VE 9 hosts (Part 1), each running VMs for a k0s cluster: 3 controllers (one per host, embedded etcd) and 5 workers (Part 2)
  • pve-03 is the small host: 7.6 GB RAM, a 2 GiB controller and a worker reserved for quorum members (Part 8)
  • Admin access to everything goes over Tailscale (Part 4)
  • A systemd timer on each host runs the power schedule:
vm-schedule stop    # 22:00: workers, other VMs, then the controller
vm-schedule start   # 06:00: controller first, 60s pause, then the rest

Issue 1: It Looked Like Tailscale Was Broken

Symptom

At 06:51, kubectl through the Tailscale operator timed out. So did SSH to pve-01 by its tailnet name, and tailscale ping pve-01:

ping "100.96.193.48" timed out

Tailscale showed the cluster’s devices reachable only through relays.

Cause

Not Tailscale. The counter-test was to reach the same machines over the LAN, skipping the tailnet:

nc -z -w4 192.168.101.30 22    # controller-01 on pve-01: timed out
nc -z -w4 192.168.101.40 22    # worker on pve-01: timed out
ssh [email protected] uptime # worker on pve-02: ok

Everything on pve-01 was unreachable over both paths, and everything elsewhere worked. Tailscale only looked broken because its paths ran through pve-01 or through the struggling cluster.

Fix

None needed for Tailscale. The real fault was a host.

Issue 2: pve-01’s NIC Stopped Without a Word

Symptom

From pve-02, the Proxmox cluster had lost one member:

Quorate:          Yes
Total votes:      2

pve-01 didn’t answer ping. TCP connections to it half-opened and then hung. After a power cycle, its journal from the previous boot showed the host had been running the whole time:

06:46:53 corosync[1250]: [KNET  ] link: host: 3 link: 0 is down
06:46:53 corosync[1250]: [KNET  ] link: host: 2 link: 0 is down
06:46:53 corosync[1250]: [KNET  ] host: host: 3 has no active links
...
06:56:57 tailscaled[892]: netcheck: UDP is blocked, trying HTTPS

The kernel log for the same period: nothing. No link-down, no NIC reset, no error.

Cause

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0e)
driver: r8169

This chip on the in-kernel r8169 driver is known for exactly this failure: the card stalls under load while the interface stays UP, often linked to power-saving features (Energy-Efficient Ethernet and PCIe ASPM). The stall came about half an hour after four VMs on that host started at once.

The host had also reset abruptly the previous evening (the journal simply stops, no shutdown sequence), which looks like a power blip rather than the same bug.

Fix

All three hosts turned out to have the same chip, so all three got both changes, staged in config and applied with staggered reboots (22:15, 22:30, 22:45) so the Proxmox cluster never dropped below 2 of 3 votes:

# /etc/network/interfaces, on the vmbr0 stanza
post-up /usr/sbin/ethtool --set-eee enp1s0f0 eee off || true

# /etc/default/grub.d/pcie-aspm.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT pcie_aspm=off"
update-grub

If it happens again, the next step is Realtek’s own r8168 driver.

Issue 3: One Slow etcd Member

Symptom

pve-01 came back, its VMs started, and yet the cluster stayed sick:

  • three of five workers NotReady, including one on a healthy host
  • cilium-operator in CrashLoopBackOff: Leader election lost
  • konnectivity agents restarting every few minutes
  • the Kyverno webhook timing out: dial tcp 10.110.202.2:443: i/o timeout

The controller’s log said where the pain was:

30 apply request took too long
15 etcdserver: request timed out
 1 lost leader

Dead end: the network again

After Issue 2, the obvious suspect was networking. But the API answered readyz in about 150 ms between the timeouts, and pod-to-pod traffic worked.

Cause

SSH to controller-03 timed out during the banner exchange, which means the VM was alive but barely responding. On its host, pve-03:

wa=8% si=0 so=0 bi=411768
VM 400 read 415 MB/s
VM 401 read 0 MB/s

The controller VM had 2 GiB of RAM and no swap (cloud images ship without it). As the API server grew (more CRDs, more objects), memory ran out. Without swap, the kernel can only evict file-backed pages, so etcd, the API server and the controllers kept re-reading their own binaries and data from disk. With embedded etcd, one slow member slows every write, and everything that depends on the API (leases, webhooks, konnectivity) times out with it.

Fix

Rebalance pve-03 without adding load to the host:

VMBeforeAfter
k0s-controller-032 GiB3 GiB (same as the other controllers)
k0s-pve-03-worker-12.5 GiB2 GiB (it runs one OpenBao pod plus DaemonSets)
qm set 400 --memory 3072 && qm reboot 400

Disk reads dropped to zero, etcd stopped logging timeouts, cilium-operator, konnectivity and Kyverno recovered. The VMs are managed by Terraform since Part 7, so vms.tf got the same values, and the plan confirmed No changes.

Side Notes

  • A failure policy decision paid off. Kyverno runs with failurePolicy: Ignore. During Issue 3, its webhook was unreachable; with Fail, every change in the cluster would have been rejected while I was trying to fix it.
  • The power schedule had its own bug, caught early. The first dry run printed nothing on two of three hosts. A helper ended with [ "$role" = "$r" ] && echo "$id", so it returned non-zero when the last VM didn’t match, and set -e quietly ended the script. Fixed with an explicit if.
  • Unexplained API noise. Before the reset, pve-01 logged authentication failure: invalid token value! every 30 seconds. Something still calls the Proxmox API with a dead token. Not related, but on the list.

Lessons

  1. Counter-test the obvious suspect. “Tailscale is down” took one nc over the LAN to disprove.
  2. A NIC can fail without a single kernel message. When a host goes quiet but the interface is UP, read the cluster’s view (corosync), not just the host’s.
  3. A control plane is only as fast as its slowest etcd member. Size controllers for the API you’ll have, not the one you started with.
  4. No swap changes how memory pressure looks. It shows up as disk reads, not swap usage.
  5. Make failure modes explicit. failurePolicy: Ignore was a conscious availability choice, and it mattered on the worst morning.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet (this post)
  11. Part 11: Hardening the Cluster in Five Layers

References