Homelab Rebuild Part 18: When the API Server Outgrew Its VM

| October 14, 2026 | 7 min to read

Intro

While I was adding a Postgres database for Part 17, one server-side dry-run failed with a webhook error that went away on retry. I almost ignored it. It turned out to be the visible edge of a bigger problem: one of my three k0s controllers was out of memory, and k0s was reacting by restarting an agent on every node.

TL;DR: kube-apiserver used 2.4 GB on a 3 GiB controller. With 99 MiB left and no swap, the VM thrashed (load 15), lost its k0s lease, and k0s rolled the konnectivity agents on every node because the controller count dropped to 2. A restart of k0scontroller through the QEMU guest agent brought it back. The lasting fix was 4 GiB on controller-01 and controller-02, applied by hand one VM at a time so etcd kept quorum, then written into Terraform so the plan came out as “No changes”.

The Situation

k0s with three controllers (etcd and kube-apiserver on each) and five workers, on three Proxmox hosts
Controllers: 3 GiB RAM, no swap, 2 cores
- controller-01 on pve-01, controller-02 on pve-02, controller-03 on pve-03
pve-03 is the small host: 7.8 GiB total, about 1.2 GiB available
The cluster runs Cilium, Kyverno, Tetragon, Flux, CloudNativePG, External Secrets, cert-manager and more: 123 CRDs in total
VMs are managed with Terraform (bpg/proxmox), applied by CI on every push to main

Issue 1: A webhook that was briefly unreachable

Symptom

Error from server (InternalError): error when creating "STDIN": Internal error occurred:
failed calling webhook "validate.externalsecret.external-secrets.io": failed to call webhook:
Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1-externalsecret?timeout=5s":
dial tcp 10.111.53.132:443: connect: no route to host

The webhook pod was running and its Service had an endpoint. Three retries a few seconds later all passed.

Cause

The API server reaches webhooks inside the cluster through konnectivity: the controllers aren’t on the pod network, so an agent on each worker tunnels the traffic. At that moment, every konnectivity agent was between 45 seconds and 5 minutes old, and one was still terminating.

The DaemonSet told the rest of the story:

gen=48
{"name":"K0S_CONTROLLER_COUNT","value":"2"}

There are three controllers, but k0s counted two. k0s writes the number of live controllers (counted from their leases) into the konnectivity DaemonSet, so every time a controller drops out or comes back, every agent restarts. Generation 48, two days after the cluster was built, meant this had happened many times. The leases showed which controller was missing:

k0s-ctrl-k0s-controller-01   2026-10-14T08:53:45.561838Z
k0s-ctrl-k0s-controller-02   2026-10-14T08:53:45.908115Z
k0s-ctrl-k0s-controller-03   2026-10-14T08:46:53.512451Z

Fix

None for the webhook itself; it was a symptom. The real issue was controller-03.

Issue 2: kube-apiserver used most of the VM

Symptom

SSH to controller-03 timed out during the banner exchange. Proxmox showed the VM almost at its limit:

qm status 400 --verbose | grep -E '^(mem|maxmem):'
# maxmem: 3221225472
# mem: 2949894144

The QEMU guest agent still answered, which gave me a way in without SSH:

qm guest exec 400 -- sh -c "uptime; free -m; ps -eo rss,comm --sort=-rss | head -3"
# load average: 15.09, 15.26, 8.22
# Mem:  2914 total, 2814 used, 99 free, 27 buff/cache, 99 available
# Swap: 0
# 2369268 kube-apiserver
#  180260 etcd

No process had been OOM-killed. The kernel was just short of page cache, so everything that touched the disk waited. That is enough to miss lease renewals.

The other two controllers weren’t far behind:

Controller	kube-apiserver RSS	Available
controller-01	1.82 GB	349 MiB
controller-02	1.88 GB	462 MiB
controller-03	2.37 GB	99 MiB

Cause

Not the amount of data. The objects in the API server were small:

apiserver_storage_objects{resource="events"} 879
apiserver_storage_objects{resource="policyreports.wgpolicyk8s.io"} 303
apiserver_storage_objects{resource="customresourcedefinitions.apiextensions.k8s.io"} 123
apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} 1.99e+08

etcd held 199 MB. The likely base cost is the 123 CRDs: kube-apiserver keeps a watch cache and the OpenAPI schema for every one of them, and the big operators (Cilium, Kyverno, Flux, CNPG, External Secrets) ship large schemas. On a 3 GiB VM, 1.8 GB of steady-state API server is already most of the room; any burst pushes the smallest one over.

This also corrects Part 16. There I wrote that the 2.3 GiB spike on controller-03 came from the Proxmox split. This time corosync on pve-01 and pve-03 logged zero token losses since 14:00, so there was no split. The split may have made it worse, but the controller was too small either way.

Fix (immediate)

Restart k0s on controller-03 through the guest agent. etcd keeps quorum with the other two members; k0s etcd member-list on controller-01 still answered, so controller-01 and controller-02 were serving:

qm guest exec 400 --timeout 150 -- systemctl restart k0scontroller

Ninety seconds later:

active
Mem: 2914 total, 1717 used, 1197 available
1211688 kube-apiserver

The API server was already at 1.2 GB and growing, so the restart only bought time.

Issue 3: Adding RAM without losing etcd

pve-01 and pve-02 each had more than 10 GiB available; pve-03 had about 1.2 GiB. So controller-01 and controller-02 could grow, and controller-03 couldn’t.

The obvious way, and why I didn’t use it

The VMs are in Terraform, and CI applies every push to main. Changing memory = 3072 to 4096 for two controllers and merging would have worked, but the provider can update and reboot both VMs in parallel. Two of three etcd members down at once means no quorum, so the API would stop until both came back.

Fix

By hand, one VM at a time. Before each step I checked that the API server reported etcd healthy, and that k0s counted three controllers again:

qm set 200 --memory 4096
qm reboot 200 --timeout 180
# rebooted 09:08:55, k0scontroller active 09:09:12
qm guest exec 200 -- free -m
# Mem: 3922

[+]etcd ok
[+]etcd-readiness ok
{"name":"K0S_CONTROLLER_COUNT","value":"3"}
daemon set "konnectivity-agent" successfully rolled out

Then controller-02 (VM 300) the same way: rebooted at 09:11:21, k0s active 18 seconds later, all three leases fresh by 09:12.

Only after that did the Terraform change go in, so the code matched what was already running. The pull request’s plan confirmed it:

No changes. Your infrastructure matches the configuration.

The apply on main reported Resources: 0 added, 0 changed, 0 destroyed.

Result

Controller	RAM	Available after
controller-01	3 GiB to 4 GiB	349 MiB to 2394 MiB
controller-02	3 GiB to 4 GiB	462 MiB to 2369 MiB
controller-03	3 GiB (unchanged)	1197 MiB right after the restart, growing

controller-03 is still the weak point. If it starts thrashing again, the recovery is the guest agent restart above. A real fix means freeing RAM on pve-03 or cutting what the API server holds, for example fewer CRDs.

Lessons

A transient webhook error can be a control plane symptom. “No route to host” from the API server to a healthy pod pointed at konnectivity, and konnectivity pointed at a controller dropping out.
k0s restarts every konnectivity agent when the controller count changes. A DaemonSet generation in the dozens is a cheap signal that a controller keeps flapping.
Budget for CRDs on the control plane. A small cluster with many operators can still need about 2 GB just for kube-apiserver.
Keep a way in that doesn’t need the guest’s network or SSH. The QEMU guest agent worked when SSH timed out.
Infrastructure as code can reboot more than you want at once. For quorum members, change them one at a time by hand, then make the code match and check for an empty plan.
Correct your own earlier conclusions. Part 16 blamed the split for the memory spike; with more data, the VM size was the real problem.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 18: When the API Server Outgrew Its VM

Intro

The Situation

Issue 1: A webhook that was briefly unreachable

Symptom

Cause

Fix

Issue 2: kube-apiserver used most of the VM

Symptom

Cause

Fix (immediate)

Issue 3: Adding RAM without losing etcd

The obvious way, and why I didn’t use it

Fix

Result

Lessons

Series

References