Homelab Rebuild Part 15: Requests and Limits for Everything

Intro

After a night of pve-01 losing power (Part 13) and a new monitoring stack (Part 14), I wanted the cluster to be predictable: every pod with a known footprint, so the scheduler places it sensibly and no single pod can eat a node. I had also looked at the descheduler for spreading pods, but it balances by requests, so requests had to come first.

This post covers the rules I used, how I rolled them out across about 20 Helm releases and manifests, and the five things that went wrong on the way.

TL;DR: Memory request about current use, memory limit about 2x, and a generous CPU limit only to stop runaways. No CPU limit on the Cilium datapath or Tetragon. A LimitRange in every app namespace catches containers a chart doesn’t expose. Along the way, Helm 4 got stuck on a post-upgrade hook, hit a server-side apply conflict on a webhook field, and a kubectl apply -f without a namespace quietly created a second Postgres cluster in default.

The Situation

• k0s cluster, five workers over three Proxmox hosts; the pve-03 worker is quorum-only (dedicated=quorum:NoSchedule)
• Helm 4.2.1 on my laptop, which uses server-side apply by default
• Most apps are Helm releases with their values in Git; a few (local-path-provisioner, cloudflared, the CNPG Cluster) are plain manifests

Before the change, from kubectl top and the pod specs:

• 17 containers were BestEffort (no requests, no limits): cert-manager (3), the CNPG operator, external-dns, external-secrets (3), Headlamp, the OpenBao server and injector, Pi-hole, the Tailscale operator, the Cilium operator, Hubble relay and UI, cilium-envoy, plus k0s’s konnectivity and nllb
• The Tailscale proxies requested 1m CPU and 1Mi memory (the operator default) while using 40 to 100 MiB
• The Kyverno cleanup controller sat at 117 MiB of a 128 MiB limit. The new “Container near memory limit” alert fired on its first day, at 91%
• Only KEDA (chart default, 1 CPU and 1000 MiB) and the Tetragon operator had CPU limits

The Rules

A CPU limit throttles; it never kills. Set tight, it slows a pod down exactly when it’s busy. So the CPU limits here are high on purpose: they only stop a runaway from starving a node.

What I left alone:

• cilium-agent, cilium-envoy and Tetragon get no CPU limit. Throttling the datapath stalls networking for every pod on the node, and throttling Tetragon drops security events.
• k0s-managed pods (coredns, metrics-server, konnectivity, nllb) stay as k0s ships them. k0s reconciles its own manifests and reverts edits.

The result is Burstable everywhere. Guaranteed QoS needs requests equal to limits, which would reserve far more CPU than these pods ever use.

The safety net

Some containers aren’t exposed in a chart’s values: Grafana’s sidecars, the Tailscale proxy’s sysctler init container. A LimitRange in each of the 16 app namespaces (not kube-system) gives them defaults. Anything set explicitly in the values wins.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-resources
  namespace: monitoring
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 10m
        memory: 64Mi
      default:
        cpu: 200m
        memory: 256Mi

Setup

Values in Git, then helm upgrade

Every change went into the chart values in Git first, then one helm upgrade per release. A kustomize server-side apply on top of a Helm release conflicts with the helm field manager, so Helm stays the only writer.

Before upgrading, I diffed helm get values against the repo for each release. An earlier pass had found a value that only existed in the live release (the Tailscale operator’s apiServerProxyConfig.mode), and an upgrade from Git would have dropped it. This time there were none.

Kyverno, for example, gained explicit resources for its admission and cleanup controllers:

admissionController:
  container:
    resources:
      requests: {cpu: 100m, memory: 192Mi}
      limits: {cpu: 500m, memory: 384Mi}
cleanupController:
  resources:
    requests: {cpu: 10m, memory: 128Mi}
    limits: {cpu: 200m, memory: 256Mi}

Tailscale proxies

The proxies are created by the operator, not by a chart, so their resources come from a ProxyClass. I created a ProxyClass named default and set it as proxyConfig.defaultProxyClass in the operator values. The proxies restarted with 10m/128Mi requests and 500m/256Mi limits.

Cilium without restarting the agent

The Cilium upgrade only touched the operator and Hubble. The agent DaemonSet generation stayed at 7, so no agent restarted and no node lost networking.

Issue 1: A chart repository that timed out

Symptom

charts.external-secrets.io ... dial tcp 74.125.68.121:443 (timeout)

Cause

The chart site was unreachable from my network at the time.

Fix

The same chart is published as an OCI artifact on GitHub’s registry:

helm upgrade external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --version <same version> -n external-secrets -f values.yaml

Issue 2: Kyverno stuck in pending-upgrade

Symptom

The Kyverno release had been failed since 22:33 the night before (pve-01 lost power in the middle of an upgrade, “context canceled”). On the re-run, the pods rolled within seconds, but helm upgrade --wait stayed in pending-upgrade past its 8 minute timeout. Only after that did the post-upgrade hook start (its kyverno-migrate-resources ServiceAccount appeared).

Cause

--wait waited for something that never reported ready, and the hook only runs after the wait. A release in pending-upgrade also blocks any further upgrade.

Fix

I stopped the upgrade, deleted the stuck revision record and the half-created hook ServiceAccount, then re-ran it with Helm 4’s --wait=hookOnly, which waits for hooks but not for every resource:

kubectl -n kyverno delete secret sh.helm.release.v1.kyverno.v3
kubectl -n kyverno delete serviceaccount kyverno-migrate-resources
helm upgrade kyverno kyverno --repo https://kyverno.github.io/kyverno/ \
  -n kyverno -f values.yaml --wait=hookOnly
# deployed in seconds

Issue 3: OpenBao and a field the injector owns

Symptom

conflict with "vault-k8s" ... .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle

The pods had already rolled, but the release was marked failed.

Cause

The OpenBao agent injector generates its own certificate and patches the caBundle of its MutatingWebhookConfiguration. With server-side apply, that field belongs to the vault-k8s field manager, and Helm 4 refuses to take it over.

Fix

Client-side apply for this release:

helm upgrade openbao openbao --repo https://openbao.github.io/openbao-helm \
  -n openbao -f values.yaml --server-side=false

It deployed, and the caBundle stayed intact. The OpenBao server StatefulSet uses OnDelete, so the new resources only apply when a pod is recreated. I deleted openbao-0, -1 and -2 one at a time, waiting for each to come back ready and unsealed (static auto-unseal, about 16 seconds each).

Issue 4: An apply that went to the wrong namespace

Symptom

The CNPG Cluster is a plain manifest. kubectl apply --server-side -f cluster.yaml printed serverside-applied, but the CPU limit never appeared on databases/pg, and CNPG didn’t roll anything.

Fix (at the time)

A merge patch on the real cluster went through:

kubectl -n databases patch cluster pg --type merge \
  -p '{"spec":{"resources":{"limits":{"cpu":"1"}}}}'

CNPG rolled both instances one at a time, the primary stayed on pg-2, and the cluster was healthy at 08:10.

Cause (found three hours later)

The first version of this post said I didn’t know why the apply did nothing. It did something: it created a second cluster. cluster.yaml has no namespace:. The kustomization next to it sets namespace: databases, and the README says kubectl apply -k. I used -f on the file alone, so the Cluster landed in default.

kubectl get cluster -A
# databases   pg   36h   2   2   Cluster in healthy state   pg-2
# default     pg   3h6m  2   2   Cluster in healthy state   pg-1

CNPG did exactly what it was told: two new, empty Postgres instances on the data-disk workers with a 20 GiB volume each, for three hours. Nothing alerted, because every check I ran after the rollout was scoped to a namespace or looked for unhealthy pods, and these were perfectly healthy.

Fix

• Deleted default/pg and its two PVCs.
• Added namespace: databases to cluster.yaml itself, so -f and -k end up in the same place.

Issue 5: The one I missed

cloudflared is a plain manifest, not a Helm release, and the first pass skipped it. A final check for containers without a CPU limit found it. It got 200m and a normal rollout.

Result

At 08:10, all nodes Ready, no pods outside Running or Completed.

The 15 BestEffort pods are cilium-envoy, konnectivity and nllb, one each per node. Envoy has no requests in the Cilium chart defaults either.

Node memory from kubectl top after the rollout:

Lessons

1. Requests come before any balancing. The scheduler and the descheduler both work from requests. Without them, a node looks empty when it isn’t.
2. CPU limits are for runaways, not for sizing. A tight CPU limit throttles a pod when it’s busiest. Memory is where the limit has to be close to real.
3. Leave the datapath alone. Throttling the CNI or the security agent hurts every pod on the node.
4. A LimitRange catches what charts hide. Sidecars and init containers get sane defaults without patching charts.
5. Diff live values before upgrading from Git. A value that only exists in the live release disappears on the next upgrade.
6. Server-side apply needs a fallback. Fields owned by another manager (a webhook’s caBundle) need --server-side=false. A stuck hook needs --wait=hookOnly.
7. Check where the object landed, not just the output. A manifest without namespace: goes to default, and serverside-applied there looks the same as an update. kubectl get <kind> -A would have shown it in seconds.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything (this post)
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google

References

• Kubernetes: Resource Management for Pods and Containers
• Kubernetes: Pod Quality of Service Classes
• Kubernetes: Limit Ranges
• Kubernetes: Server-Side Apply
• Helm: helm upgrade
• Tailscale: Customizing the operator’s resources with ProxyClass
• CloudNativePG: Resource management