Homelab Rebuild Part 18: When the API Server Outgrew Its VM

| October 14, 2026 | 8 min to read

Intro

While I added a Postgres database for Part 17, one server-side dry-run failed with a webhook error. The error went away on retry. I almost ignored it. It was a symptom of a bigger problem. One of my three k0s controllers was out of memory. k0s reacted to that with a restart of an agent on every node.

TL;DR: kube-apiserver used 2.4 GB on a 3 GiB controller. With 99 MiB left and no swap, the VM thrashed (load 15) and lost its k0s lease. The controller count dropped to 2, so k0s rolled the konnectivity agents on every node. A restart of k0scontroller through the QEMU guest agent brought it back. The lasting fix was 4 GiB on controller-01 and controller-02. I applied it by hand one VM at a time, so etcd kept quorum. Then I wrote it into Terraform, and the plan came out as “No changes”.

The Situation

k0s with three controllers (etcd and kube-apiserver on each) and five workers, on three Proxmox hosts
Controllers: 3 GiB RAM, no swap, 2 cores
- controller-01 on pve-01, controller-02 on pve-02, controller-03 on pve-03
pve-03 is the small host: 7.8 GiB total, about 1.2 GiB available
The cluster runs Cilium, Kyverno, Tetragon, Flux, CloudNativePG, External Secrets, cert-manager and more: 123 CRDs in total
Terraform (bpg/proxmox) manages the VMs. CI applies it on every push to main

Issue 1: A webhook that was briefly unreachable

Symptom

Error from server (InternalError): error when creating "STDIN": Internal error occurred:
failed calling webhook "validate.externalsecret.external-secrets.io": failed to call webhook:
Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1-externalsecret?timeout=5s":
dial tcp 10.111.53.132:443: connect: no route to host

The webhook pod was running, and its Service had an endpoint. A few seconds later, three retries all passed.

Cause

The API server reaches webhooks inside the cluster through konnectivity. The controllers are not on the pod network, so an agent on each worker tunnels the traffic. At that moment, every konnectivity agent was between 45 seconds and 5 minutes old, and one was still terminating.

The DaemonSet told the rest of the story:

gen=48
{"name":"K0S_CONTROLLER_COUNT","value":"2"}

There are three controllers, but k0s counted two. k0s writes the number of live controllers (counted from their leases) into the konnectivity DaemonSet. So each time a controller drops out or comes back, every agent restarts. Generation 48, two days after I built the cluster, meant that this happened many times. The leases showed which controller was missing:

k0s-ctrl-k0s-controller-01   2026-10-14T08:53:45.561838Z
k0s-ctrl-k0s-controller-02   2026-10-14T08:53:45.908115Z
k0s-ctrl-k0s-controller-03   2026-10-14T08:46:53.512451Z

Fix

None for the webhook itself. It was a symptom. The real issue was controller-03.

Issue 2: kube-apiserver used most of the VM

Symptom

SSH to controller-03 timed out during the banner exchange. Proxmox showed the VM almost at its limit:

qm status 400 --verbose | grep -E '^(mem|maxmem):'
# maxmem: 3221225472
# mem: 2949894144

The QEMU guest agent still answered. That gave me a way in without SSH:

qm guest exec 400 -- sh -c "uptime; free -m; ps -eo rss,comm --sort=-rss | head -3"
# load average: 15.09, 15.26, 8.22
# Mem:  2914 total, 2814 used, 99 free, 27 buff/cache, 99 available
# Swap: 0
# 2369268 kube-apiserver
#  180260 etcd

The OOM killer did not kill any process. The kernel was short of page cache, so everything that touched the disk waited. That is enough to miss lease renewals.

The other two controllers were not far behind:

Controller	kube-apiserver RSS	Available
controller-01	1.82 GB	349 MiB
controller-02	1.88 GB	462 MiB
controller-03	2.37 GB	99 MiB

Cause

The cause was not the amount of data. The objects in the API server were small:

apiserver_storage_objects{resource="events"} 879
apiserver_storage_objects{resource="policyreports.wgpolicyk8s.io"} 303
apiserver_storage_objects{resource="customresourcedefinitions.apiextensions.k8s.io"} 123
apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} 1.99e+08

etcd held 199 MB. The likely base cost is the 123 CRDs. kube-apiserver keeps a watch cache and the OpenAPI schema for every one of them. The big operators (Cilium, Kyverno, Flux, CNPG, External Secrets) ship large schemas. On a 3 GiB VM, 1.8 GB of steady-state API server already takes most of the room. Any burst pushes the smallest one over.

This also corrects Part 16. There I wrote that the 2.3 GiB spike on controller-03 came from the Proxmox split. This time corosync on pve-01 and pve-03 logged zero token losses since 14:00, so there was no split. The split possibly made it worse, but the controller was too small either way.

Fix (immediate)

Restart k0s on controller-03 through the guest agent. etcd keeps quorum with the other two members. k0s etcd member-list on controller-01 still answered, so controller-01 and controller-02 were serving:

qm guest exec 400 --timeout 150 -- systemctl restart k0scontroller

Ninety seconds later:

active
Mem: 2914 total, 1717 used, 1197 available
1211688 kube-apiserver

The API server was already at 1.2 GB and growing, so the restart only bought time.

Issue 3: More RAM without losing etcd

pve-01 and pve-02 each had more than 10 GiB available. pve-03 had about 1.2 GiB. So controller-01 and controller-02 could grow, and controller-03 could not.

The obvious way, and why I did not use it

The VMs are in Terraform, and CI applies every push to main. I can change memory = 3072 to 4096 for two controllers and merge. That works, but the provider can update and reboot both VMs in parallel. If two of three etcd members are down at once, there is no quorum. The API then stops until both come back.

Fix

Do it by hand, one VM at a time. Before each step I made sure that the API server reported etcd healthy and that k0s counted three controllers again:

qm set 200 --memory 4096
qm reboot 200 --timeout 180
# rebooted 09:08:55, k0scontroller active 09:09:12
qm guest exec 200 -- free -m
# Mem: 3922

[+]etcd ok
[+]etcd-readiness ok
{"name":"K0S_CONTROLLER_COUNT","value":"3"}
daemon set "konnectivity-agent" successfully rolled out

Then I did controller-02 (VM 300) the same way. It rebooted at 09:11:21, and k0s was active 18 seconds later. All three leases were fresh by 09:12.

Only after that did the Terraform change go in, so the code matched what was already running. The plan of the pull request confirmed it:

No changes. Your infrastructure matches the configuration.

The apply on main reported Resources: 0 added, 0 changed, 0 destroyed.

Result

Controller	RAM	Available after
controller-01	3 GiB to 4 GiB	349 MiB to 2394 MiB
controller-02	3 GiB to 4 GiB	462 MiB to 2369 MiB
controller-03	3 GiB (unchanged)	1197 MiB right after the restart, growing

controller-03 is still the weak point. If it starts to thrash again, the recovery is the guest agent restart above. A real fix needs free RAM on pve-03, or less data in the API server, for example fewer CRDs.

Lessons

A transient webhook error can be a control plane symptom. “No route to host” from the API server to a healthy pod pointed at konnectivity. Then konnectivity pointed at a controller that dropped out.
k0s restarts every konnectivity agent when the controller count changes. A DaemonSet generation in the dozens is a cheap signal that a controller keeps flapping (dropping out and coming back).
Budget for CRDs on the control plane. A small cluster with many operators can still need about 2 GB only for kube-apiserver.
Keep a way in that does not need the network of the guest or SSH. The QEMU guest agent worked when SSH timed out.
Infrastructure as code can reboot more than you want at once. For quorum members, change them one at a time by hand. Then make the code match and check for an empty plan.
Correct your own earlier conclusions. Part 16 blamed the split for the memory spike. With more data, the VM size was the real problem.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 18: When the API Server Outgrew Its VM

Intro

The Situation

Issue 1: A webhook that was briefly unreachable

Symptom

Cause

Fix

Issue 2: kube-apiserver used most of the VM

Symptom

Cause

Fix (immediate)

Issue 3: More RAM without losing etcd

The obvious way, and why I did not use it

Fix

Result

Lessons

Series

References