Intro
To cut electricity costs, I added a nightly power schedule. Every VM shuts down
at 22:00 and starts again at 06:00. The next morning everything came back. Half
an hour later, kubectl stopped answering.
Two separate problems followed, one on top of the other. There was also one red herring (a false lead) that pointed at the wrong component.
TL;DR: The Realtek RTL8111/8168 NIC (
r8169driver) on one Proxmox host stopped passing traffic with no kernel error, and the link stayedUP. All three hosts had the same chip, so all three got EEE off andpcie_aspm=off. After recovery, a controller VM with 2 GiB and no swap thrashed its disk at 415 MB/s. That one slow etcd member degraded the entire control plane.
The Situation
- Three Proxmox VE 9 hosts (Part 1), each running VMs for a k0s cluster: 3 controllers (one per host, embedded etcd) and 5 workers (Part 2)
pve-03is the small host: 7.6 GB RAM, a 2 GiB controller and a worker reserved for quorum members (Part 8)- Admin access to everything goes over Tailscale (Part 4)
- A systemd timer on each host runs the power schedule:
vm-schedule stop # 22:00: workers, other VMs, then the controller
vm-schedule start # 06:00: controller first, 60s pause, then the rest
Issue 1: It Looked Like Tailscale Was Broken
Symptom
At 06:51, kubectl through the Tailscale operator timed out. So did SSH to
pve-01 by its tailnet name, and tailscale ping pve-01:
ping "100.96.193.48" timed out
Tailscale showed that it reached the cluster devices only through relays.
Cause
It was not Tailscale. For the counter-test, I tried to reach the same machines over the LAN, without the tailnet:
nc -z -w4 192.168.101.30 22 # controller-01 on pve-01: timed out
nc -z -w4 192.168.101.40 22 # worker on pve-01: timed out
ssh root@192.168.101.50 uptime # worker on pve-02: ok
Everything on pve-01 was unreachable over both paths. Everything else worked. Tailscale only looked broken because its paths ran through pve-01 or through the struggling cluster.
Fix
Tailscale needed no fix. The real fault was a host.
Issue 2: pve-01’s NIC Stopped Without a Word
Symptom
From pve-02, the Proxmox cluster had lost one member:
Quorate: Yes
Total votes: 2
pve-01 did not answer ping. TCP connections to it half-opened and then hung. After a power cycle, I read its journal from the previous boot. The host was running the whole time:
06:46:53 corosync[1250]: [KNET ] link: host: 3 link: 0 is down
06:46:53 corosync[1250]: [KNET ] link: host: 2 link: 0 is down
06:46:53 corosync[1250]: [KNET ] host: host: 3 has no active links
...
06:56:57 tailscaled[892]: netcheck: UDP is blocked, trying HTTPS
The kernel log for the same period: nothing. No link-down, no NIC reset, no error.
Cause
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0e)
driver: r8169
This chip on the in-kernel r8169 driver is known for exactly this failure. The
card stalls under load while the interface stays UP. The cause is often a
power-saving feature (Energy-Efficient Ethernet or PCIe ASPM). The stall came
about half an hour after four VMs on that host started at once.
The host also reset abruptly the previous evening. The journal stops with no shutdown sequence. That looks like a power blip, not the same bug.
Fix
All three hosts had the same chip, so all three got both changes. I staged them in the configuration and applied them with staggered reboots (22:15, 22:30, 22:45). That way, the Proxmox cluster never dropped below 2 of 3 votes:
# /etc/network/interfaces, on the vmbr0 stanza
post-up /usr/sbin/ethtool --set-eee enp1s0f0 eee off || true
# /etc/default/grub.d/pcie-aspm.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT pcie_aspm=off"
update-grub
If it happens again, the next step is the r8168 driver from Realtek.
Issue 3: One Slow etcd Member
Symptom
pve-01 came back and its VMs started. But the cluster stayed sick:
- three of five workers
NotReady, including one on a healthy host cilium-operatorinCrashLoopBackOff:Leader election lost- konnectivity agents restarting every few minutes
- the Kyverno webhook timing out:
dial tcp 10.110.202.2:443: i/o timeout
The log of the controller showed where the problem was:
30 apply request took too long
15 etcdserver: request timed out
1 lost leader
Dead end: the network again
After Issue 2, the obvious suspect was networking. But the API answered
readyz in about 150 ms between the timeouts, and pod-to-pod traffic worked.
Cause
SSH to controller-03 timed out during the banner exchange. That means that
the VM was alive but barely responding. On its host, pve-03:
wa=8% si=0 so=0 bi=411768
VM 400 read 415 MB/s
VM 401 read 0 MB/s
The controller VM had 2 GiB of RAM and no swap (cloud images ship without it). As the API server grew (more CRDs, more objects), memory ran out. Without swap, the kernel can only evict file-backed pages (memory that holds file contents). So etcd, the API server and the controllers kept re-reading their own binaries and data from disk. With embedded etcd, one slow member slows every write. Everything that depends on the API (leases, webhooks, konnectivity) times out with it.
Fix
Rebalance pve-03 without adding load to the host:
| VM | Before | After |
|---|---|---|
k0s-controller-03 | 2 GiB | 3 GiB (same as the other controllers) |
k0s-pve-03-worker-1 | 2.5 GiB | 2 GiB (it runs one OpenBao pod plus DaemonSets) |
qm set 400 --memory 3072 && qm reboot 400
Disk reads dropped to zero and etcd stopped logging timeouts. cilium-operator,
konnectivity and Kyverno recovered. Terraform manages the VMs since Part 7, so
vms.tf got the same values. The plan confirmed No changes.
Side Notes
- A failure policy decision paid off. Kyverno runs with
failurePolicy: Ignore. During Issue 3, its webhook was unreachable. WithFail, the cluster rejects every change, including the changes I made to fix it. - The power schedule had its own bug, caught early. The first dry run
printed nothing on two of three hosts. A helper ended with
[ "$role" = "$r" ] && echo "$id". So it returned non-zero when the last VM did not match, andset -eended the script without a word. I fixed it with an explicitif. - Unexplained API noise. Before the reset, pve-01 logged
authentication failure: invalid token value!every 30 seconds. Something still calls the Proxmox API with a dead token. It is not related, but it is on the list.
Lessons
- Counter-test the obvious suspect. One
ncover the LAN disproved “Tailscale is down”. - A NIC can fail without a single kernel message. If a host goes quiet
but the interface is
UP, read the view of the cluster (corosync), not only the view of the host. - A control plane is only as fast as its slowest etcd member. Size controllers for the API you will have, not the one you started with.
- No swap changes how memory pressure looks. It shows up as disk reads, not swap usage.
- Make failure modes explicit.
failurePolicy: Ignorewas a conscious choice for availability. It mattered on the worst morning.
Update: a lookalike that was not the NIC (2026-10-16)
Three days later, pve-02 and then pve-01 dropped off the network with the same
symptoms: interface UP, no kernel errors, corosync links down. This time the
NICs were fine. Their TX counters kept moving, and the gateway took 5.5 seconds
to answer from every host. The Wi-Fi router, which is also the only switch in
my LAN, choked. A restart fixed it. Before blaming the NIC, check the
TX counter twice and ping the gateway from two hosts. Details in
Part 25.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet (this post)
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed