Back to Home

Homelab Rebuild Part 10: When the Hypervisor Went Quiet

| October 13, 2026 | 8 min to read

Intro

To cut electricity costs, I added a nightly power schedule. Every VM shuts down at 22:00 and starts again at 06:00. The next morning everything came back. Half an hour later, kubectl stopped answering.

Two separate problems followed, one on top of the other. There was also one red herring (a false lead) that pointed at the wrong component.

TL;DR: The Realtek RTL8111/8168 NIC (r8169 driver) on one Proxmox host stopped passing traffic with no kernel error, and the link stayed UP. All three hosts had the same chip, so all three got EEE off and pcie_aspm=off. After recovery, a controller VM with 2 GiB and no swap thrashed its disk at 415 MB/s. That one slow etcd member degraded the entire control plane.

The Situation

  • Three Proxmox VE 9 hosts (Part 1), each running VMs for a k0s cluster: 3 controllers (one per host, embedded etcd) and 5 workers (Part 2)
  • pve-03 is the small host: 7.6 GB RAM, a 2 GiB controller and a worker reserved for quorum members (Part 8)
  • Admin access to everything goes over Tailscale (Part 4)
  • A systemd timer on each host runs the power schedule:
vm-schedule stop    # 22:00: workers, other VMs, then the controller
vm-schedule start   # 06:00: controller first, 60s pause, then the rest

Issue 1: It Looked Like Tailscale Was Broken

Symptom

At 06:51, kubectl through the Tailscale operator timed out. So did SSH to pve-01 by its tailnet name, and tailscale ping pve-01:

ping "100.96.193.48" timed out

Tailscale showed that it reached the cluster devices only through relays.

Cause

It was not Tailscale. For the counter-test, I tried to reach the same machines over the LAN, without the tailnet:

nc -z -w4 192.168.101.30 22    # controller-01 on pve-01: timed out
nc -z -w4 192.168.101.40 22    # worker on pve-01: timed out
ssh root@192.168.101.50 uptime # worker on pve-02: ok

Everything on pve-01 was unreachable over both paths. Everything else worked. Tailscale only looked broken because its paths ran through pve-01 or through the struggling cluster.

Fix

Tailscale needed no fix. The real fault was a host.

Issue 2: pve-01’s NIC Stopped Without a Word

Symptom

From pve-02, the Proxmox cluster had lost one member:

Quorate:          Yes
Total votes:      2

pve-01 did not answer ping. TCP connections to it half-opened and then hung. After a power cycle, I read its journal from the previous boot. The host was running the whole time:

06:46:53 corosync[1250]: [KNET  ] link: host: 3 link: 0 is down
06:46:53 corosync[1250]: [KNET  ] link: host: 2 link: 0 is down
06:46:53 corosync[1250]: [KNET  ] host: host: 3 has no active links
...
06:56:57 tailscaled[892]: netcheck: UDP is blocked, trying HTTPS

The kernel log for the same period: nothing. No link-down, no NIC reset, no error.

Cause

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0e)
driver: r8169

This chip on the in-kernel r8169 driver is known for exactly this failure. The card stalls under load while the interface stays UP. The cause is often a power-saving feature (Energy-Efficient Ethernet or PCIe ASPM). The stall came about half an hour after four VMs on that host started at once.

The host also reset abruptly the previous evening. The journal stops with no shutdown sequence. That looks like a power blip, not the same bug.

Fix

All three hosts had the same chip, so all three got both changes. I staged them in the configuration and applied them with staggered reboots (22:15, 22:30, 22:45). That way, the Proxmox cluster never dropped below 2 of 3 votes:

# /etc/network/interfaces, on the vmbr0 stanza
post-up /usr/sbin/ethtool --set-eee enp1s0f0 eee off || true
# /etc/default/grub.d/pcie-aspm.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT pcie_aspm=off"
update-grub

If it happens again, the next step is the r8168 driver from Realtek.

Issue 3: One Slow etcd Member

Symptom

pve-01 came back and its VMs started. But the cluster stayed sick:

  • three of five workers NotReady, including one on a healthy host
  • cilium-operator in CrashLoopBackOff: Leader election lost
  • konnectivity agents restarting every few minutes
  • the Kyverno webhook timing out: dial tcp 10.110.202.2:443: i/o timeout

The log of the controller showed where the problem was:

30 apply request took too long
15 etcdserver: request timed out
 1 lost leader

Dead end: the network again

After Issue 2, the obvious suspect was networking. But the API answered readyz in about 150 ms between the timeouts, and pod-to-pod traffic worked.

Cause

SSH to controller-03 timed out during the banner exchange. That means that the VM was alive but barely responding. On its host, pve-03:

wa=8% si=0 so=0 bi=411768
VM 400 read 415 MB/s
VM 401 read 0 MB/s

The controller VM had 2 GiB of RAM and no swap (cloud images ship without it). As the API server grew (more CRDs, more objects), memory ran out. Without swap, the kernel can only evict file-backed pages (memory that holds file contents). So etcd, the API server and the controllers kept re-reading their own binaries and data from disk. With embedded etcd, one slow member slows every write. Everything that depends on the API (leases, webhooks, konnectivity) times out with it.

Fix

Rebalance pve-03 without adding load to the host:

VMBeforeAfter
k0s-controller-032 GiB3 GiB (same as the other controllers)
k0s-pve-03-worker-12.5 GiB2 GiB (it runs one OpenBao pod plus DaemonSets)
qm set 400 --memory 3072 && qm reboot 400

Disk reads dropped to zero and etcd stopped logging timeouts. cilium-operator, konnectivity and Kyverno recovered. Terraform manages the VMs since Part 7, so vms.tf got the same values. The plan confirmed No changes.

Side Notes

  • A failure policy decision paid off. Kyverno runs with failurePolicy: Ignore. During Issue 3, its webhook was unreachable. With Fail, the cluster rejects every change, including the changes I made to fix it.
  • The power schedule had its own bug, caught early. The first dry run printed nothing on two of three hosts. A helper ended with [ "$role" = "$r" ] && echo "$id". So it returned non-zero when the last VM did not match, and set -e ended the script without a word. I fixed it with an explicit if.
  • Unexplained API noise. Before the reset, pve-01 logged authentication failure: invalid token value! every 30 seconds. Something still calls the Proxmox API with a dead token. It is not related, but it is on the list.

Lessons

  1. Counter-test the obvious suspect. One nc over the LAN disproved “Tailscale is down”.
  2. A NIC can fail without a single kernel message. If a host goes quiet but the interface is UP, read the view of the cluster (corosync), not only the view of the host.
  3. A control plane is only as fast as its slowest etcd member. Size controllers for the API you will have, not the one you started with.
  4. No swap changes how memory pressure looks. It shows up as disk reads, not swap usage.
  5. Make failure modes explicit. failurePolicy: Ignore was a conscious choice for availability. It mattered on the worst morning.

Update: a lookalike that was not the NIC (2026-10-16)

Three days later, pve-02 and then pve-01 dropped off the network with the same symptoms: interface UP, no kernel errors, corosync links down. This time the NICs were fine. Their TX counters kept moving, and the gateway took 5.5 seconds to answer from every host. The Wi-Fi router, which is also the only switch in my LAN, choked. A restart fixed it. Before blaming the NIC, check the TX counter twice and ping the gateway from two hosts. Details in Part 25.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet (this post)
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths
  28. Part 28: Observability, Take Three: Making It Worth Looking At
  29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
  30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
  31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
  32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
  33. Part 33: The Firewall Held, the Tailnet Did Not
  34. Part 34: From Watching to Blocking, and a Record of Who Did What
  35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
  36. Part 36: The Fans Were Loud, the CPUs Were Idle
  37. Part 37: A Power Trip, a Search Domain and a Wildcard
  38. Part 38: An Autonomous AI Pentester in a Sealed Box
  39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References