Homelab Rebuild Part 15: Requests and Limits for Everything

| October 14, 2026 | 10 min to read

Intro

After a night of power loss on pve-01 (Part 13) and a new monitoring stack (Part 14), I wanted the cluster to be predictable. I wanted every pod to have a known footprint. Then the scheduler places it sensibly, and no single pod can eat a node. I also looked at the descheduler to spread pods. But it balances by requests, so requests had to come first.

This post covers the rules I used and how I rolled them out across about 20 Helm releases and manifests. It also covers the five things that went wrong on the way.

TL;DR: Memory request about current use, memory limit about 2x, and a generous CPU limit only to stop runaways. No CPU limit on the Cilium datapath or Tetragon. A LimitRange in every app namespace catches containers that a chart does not expose. Along the way, Helm 4 got stuck on a post-upgrade hook and hit a server-side apply conflict on a webhook field. Also, a kubectl apply -f without a namespace created a second Postgres cluster in default.

The Situation

k0s cluster, five workers over three Proxmox hosts. The pve-03 worker is quorum-only (dedicated=quorum:NoSchedule)
Helm 4.2.1 on my laptop. It uses server-side apply by default
Most apps are Helm releases with their values in Git. A few (local-path-provisioner, cloudflared, the CNPG Cluster) are plain manifests

Before the change, from kubectl top and the pod specs:

17 containers were BestEffort (no requests, no limits): cert-manager (3), the CNPG operator, external-dns, external-secrets (3), Headlamp, the OpenBao server and injector, Pi-hole, the Tailscale operator, the Cilium operator, Hubble relay and UI, cilium-envoy, plus konnectivity and nllb from k0s
The Tailscale proxies requested 1m CPU and 1Mi memory (the operator default), but they used 40 to 100 MiB
The Kyverno cleanup controller sat at 117 MiB of a 128 MiB limit. The new “Container near memory limit” alert fired on its first day, at 91%
Only KEDA (chart default, 1 CPU and 1000 MiB) and the Tetragon operator had CPU limits

The Rules

What	Request	Limit
Memory	About current use	About 2x the request
CPU, controllers and webhooks	Current use	`200m`
CPU, mid-size (Cilium operator, Hubble, OTel, VictoriaLogs, Grafana, Pi-hole, OpenBao server, Tailscale proxies)	Current use	`500m`
CPU, data stores (VictoriaMetrics, Postgres)	Current use	`1`
ARC runner job pods (Terraform)	Small	`2` CPU, `2Gi`

A CPU limit throttles. It never kills. If the limit is tight, it slows a pod down exactly when the pod is busy. So the CPU limits here are high on purpose. They only stop a runaway from starving a node.

What I left alone:

cilium-agent, cilium-envoy and Tetragon get no CPU limit. If the datapath is throttled, networking stalls for every pod on the node. If Tetragon is throttled, it drops security events.
Pods that k0s manages (coredns, metrics-server, konnectivity, nllb) stay as k0s ships them. k0s reconciles its own manifests and reverts edits.

The result is Burstable everywhere. Guaranteed QoS needs requests equal to limits. That reserves far more CPU than these pods ever use.

The safety net

The values of some charts do not expose all containers. Examples are the Grafana sidecars and the sysctler init container of the Tailscale proxy. A LimitRange in each of the 16 app namespaces (not kube-system) gives them defaults. Anything set explicitly in the values wins.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-resources
  namespace: monitoring
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 10m
        memory: 64Mi
      default:
        cpu: 200m
        memory: 256Mi

Setup

Values in Git, then helm upgrade

Every change went into the chart values in Git first. Then I ran one helm upgrade per release. A kustomize server-side apply on top of a Helm release conflicts with the helm field manager. So Helm stays the only writer.

Before each upgrade, I diffed helm get values against the repo for that release. An earlier pass found a value that existed only in the live release (apiServerProxyConfig.mode of the Tailscale operator). An upgrade from Git can drop such a value. This time there were none.

For example, Kyverno got explicit resources for its admission and cleanup controllers:

admissionController:
  container:
    resources:
      requests: {cpu: 100m, memory: 192Mi}
      limits: {cpu: 500m, memory: 384Mi}
cleanupController:
  resources:
    requests: {cpu: 10m, memory: 128Mi}
    limits: {cpu: 200m, memory: 256Mi}

Tailscale proxies

The operator creates the proxies, not a chart. So their resources come from a ProxyClass. I created a ProxyClass named default and set it as proxyConfig.defaultProxyClass in the operator values. The proxies restarted with 10m/128Mi requests and 500m/256Mi limits.

Cilium without an agent restart

The Cilium upgrade touched only the operator and Hubble. The generation of the agent DaemonSet stayed at 7. So no agent restarted, and no node lost networking.

Issue 1: A chart repository that timed out

Symptom

charts.external-secrets.io ... dial tcp 74.125.68.121:443 (timeout)

Cause

The chart site was unreachable from my network at the time.

Fix

The same chart is also published as an OCI artifact on the GitHub registry:

helm upgrade external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --version <same version> -n external-secrets -f values.yaml

Issue 2: Kyverno stuck in pending-upgrade

Symptom

The Kyverno release was failed since 22:33 the night before. pve-01 lost power in the middle of an upgrade (“context canceled”). On the re-run, the pods rolled within seconds. But helm upgrade --wait stayed in pending-upgrade past its 8 minute timeout. Only after that did the post-upgrade hook start (its kyverno-migrate-resources ServiceAccount appeared).

Cause

--wait waited for something that never reported ready. The hook runs only after the wait. A release in pending-upgrade also blocks any further upgrade.

Fix

I stopped the upgrade. I deleted the stuck revision record and the half-created hook ServiceAccount. Then I ran it again with --wait=hookOnly from Helm 4. That flag waits for hooks but not for every resource:

kubectl -n kyverno delete secret sh.helm.release.v1.kyverno.v3
kubectl -n kyverno delete serviceaccount kyverno-migrate-resources
helm upgrade kyverno kyverno --repo https://kyverno.github.io/kyverno/ \
  -n kyverno -f values.yaml --wait=hookOnly
# deployed in seconds

Issue 3: OpenBao and a field the injector owns

Symptom

conflict with "vault-k8s" ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle

The pods already rolled, but the release was marked failed.

Cause

The OpenBao agent injector generates its own certificate and patches the caBundle of its MutatingWebhookConfiguration. With server-side apply, that field belongs to the vault-k8s field manager. Helm 4 refuses to take it over.

Fix

Use client-side apply for this release:

helm upgrade openbao openbao --repo https://openbao.github.io/openbao-helm \
  -n openbao -f values.yaml --server-side=false

It deployed, and the caBundle stayed intact. The OpenBao server StatefulSet uses OnDelete, so the new resources apply only when a pod is recreated. I deleted openbao-0, -1 and -2 one at a time. I waited for each to come back ready and unsealed (static auto-unseal, about 16 seconds each).

Issue 4: An apply that went to the wrong namespace

Symptom

The CNPG Cluster is a plain manifest. kubectl apply --server-side -f cluster.yaml printed serverside-applied. But the CPU limit never appeared on databases/pg, and CNPG did not roll anything.

Fix (at the time)

A merge patch on the real cluster went through:

kubectl -n databases patch cluster pg --type merge \
  -p '{"spec":{"resources":{"limits":{"cpu":"1"}}}}'

CNPG rolled both instances one at a time. The primary stayed on pg-2, and the cluster was healthy at 08:10.

Cause (found three hours later)

The first version of this post said that I did not know why the apply did nothing. It did something: it created a second cluster. cluster.yaml has no namespace:. The kustomization next to it sets namespace: databases, and the README says kubectl apply -k. I used -f on the file alone, so the Cluster landed in default.

kubectl get cluster -A
# databases   pg   36h   2   2   Cluster in healthy state   pg-2
# default     pg   3h6m  2   2   Cluster in healthy state   pg-1

CNPG did exactly what I told it. It ran two new, empty Postgres instances on the data-disk workers with a 20 GiB volume each, for three hours. Nothing alerted. Every check I ran after the rollout was scoped to a namespace or looked for unhealthy pods. These pods were perfectly healthy.

Fix

Deleted default/pg and its two PVCs.
Added namespace: databases to cluster.yaml itself. Now -f and -k end up in the same place.

Issue 5: The one I missed

cloudflared is a plain manifest, not a Helm release. The first pass skipped it. A final check for containers without a CPU limit found it. It got 200m and a normal rollout.

Result

At 08:10, all nodes were Ready, and no pods were outside Running or Completed.

	Before	After
BestEffort containers outside kube-system	17	0
Pods Burstable / BestEffort		61 / 15
Containers without a CPU limit	All but KEDA and the Tetragon operator	By design only

The 15 BestEffort pods are cilium-envoy, konnectivity and nllb, one each per node. Envoy has no requests in the Cilium chart defaults either.

Node memory from kubectl top after the rollout:

Node	Memory
k0s-pve-01-worker-1	34%
k0s-pve-01-worker-2	38%
k0s-pve-02-worker-1	38%
k0s-pve-02-worker-2	45%
k0s-pve-03-worker-1	52%

Lessons

Requests come before any balancing. The scheduler and the descheduler both work from requests. Without them, a node looks empty when it is not.
CPU limits are for runaways, not for sizing. A tight CPU limit throttles a pod when it is busiest. For memory, the limit must be close to real use.
Leave the datapath alone. If you throttle the CNI or the security agent, every pod on the node suffers.
A LimitRange catches what charts hide. Sidecars and init containers get sane defaults without patches to the charts.
Diff live values before you upgrade from Git. A value that exists only in the live release disappears on the next upgrade.
Server-side apply needs a fallback. Fields that another manager owns (the caBundle of a webhook) need --server-side=false. A stuck hook needs --wait=hookOnly.
Check where the object landed, not only the output. A manifest without namespace: goes to default, and serverside-applied there looks the same as an update. kubectl get <kind> -A shows it in seconds.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 15: Requests and Limits for Everything

Intro

The Situation

The Rules

The safety net

Setup

Values in Git, then helm upgrade

Tailscale proxies

Cilium without an agent restart

Issue 1: A chart repository that timed out

Symptom

Cause

Fix

Issue 2: Kyverno stuck in pending-upgrade

Symptom

Cause

Fix

Issue 3: OpenBao and a field the injector owns

Symptom

Cause

Fix

Issue 4: An apply that went to the wrong namespace

Symptom

Fix (at the time)

Cause (found three hours later)

Fix

Issue 5: The one I missed

Result

Lessons

Series

References