Homelab Rebuild Part 11: Hardening the Cluster in Five Layers

Intro

The rebuilt cluster had Kyverno installed in audit-only mode: the Pod Security Standards as policy reports, no blocking. Time to turn that into actual protection against common misconfigurations and runtime attacks, without breaking the system components that legitimately need privileges.

TL;DR: Five layers. Pod Security Admission enforces baseline on every app namespace. Kyverno closes gaps specific to this cluster (secret scoping, Service types, exec into sensitive namespaces), audits hygiene and adds safe defaults. Tetragon watches what admission control can’t see: what processes do at runtime. Two gotchas: the Kyverno policies chart silently ignores Deny in its new CEL mode, and the Tailscale API proxy ignores kubectl --as, which made a working policy look broken.

The Situation

• Kyverno v1.19.1, chart policies (Pod Security Standards) in Audit
• Tetragon 1.7.1
• System components that need privileges: Cilium and k0s’s own pods in kube-system, Tailscale proxies, local-path’s helper pods
• Admission control sees API requests. It can stop a bad Pod spec; it can’t see a shell spawned inside a running container. That’s why layer 5 exists.

The audit reports were encouraging: of 298 would-fail results, 233 came from kube-system. My own apps were mostly clean, so enforcement could start soon, with narrow exclusions for the system namespaces.

Layer 1: Pod Security, With the Built-in Admission

Issue 1: The chart ignored Deny

My plan was to switch the chart’s baseline policies to Deny with validationFailureActionByPolicy, and exclude system namespaces with policyExclude. Rendering the chart showed neither made it into the policy:

# values.yaml
validationFailureActionByPolicy:
  disallow-host-ports: Deny
policyExclude:
  disallow-host-ports:
    any:
      - resources:
          namespaces: [kube-system, tailscale]

helm template kp kyverno-policies --repo https://kyverno.github.io/kyverno/ --version 3.9.1 -f values.yaml

# rendered disallow-host-ports: still Audit, no exclusion
validationActions:
  - Audit
matchConstraints:
  resourceRules:
    - resources: [pods]
      operations: [CREATE, UPDATE]

In the chart’s CEL mode (policyType: ValidatingPolicy), both settings are silently ignored.

Fix: Pod Security Admission

Kubernetes already ships the exact baseline standard as Pod Security Admission. One label per namespace:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest

A server-side dry run first showed whether any running pod would violate it:

kubectl label ns openbao pod-security.kubernetes.io/enforce=baseline --dry-run=server

No warnings on any of the 19 app namespaces. For namespaces created later, a Kyverno MutatingPolicy adds the label automatically (skipping kube-system, tailscale and local-path-storage). The chart’s 17 policies stay in Audit as a report of the stricter restricted level.

Layer 2: Rules Only This Cluster Needs

Issue 2: Any namespace could read every secret

External Secrets uses one cluster-wide store backed by OpenBao (Part 5). Any namespace could create an ExternalSecret pointing at any path, including the Cloudflare token. A CEL ValidatingPolicy now requires every key to sit under the namespace’s own path:

validations:
  - expression: >-
      !has(object.spec.data) ||
      object.spec.data.all(d, d.remoteRef.key.startsWith(request.namespace + "/"))
    messageExpression: >-
      "spec.data[*].remoteRef.key must start with '" + request.namespace + "/'"

It found one existing violation: external-dns read cert-manager/cloudflare. It now has its own copy at external-dns/cloudflare.

Opt-in internet exposure, natively

Apps reach the internet through a dedicated public-gateway behind a Cloudflare Tunnel (Part 9). Making that opt-in didn’t need Kyverno: Gateway API listeners can select which namespaces may attach routes.

allowedRoutes:
  namespaces:
    from: Selector
    selector:
      matchLabels:
        exposure: public

A route from an unlabeled namespace gets Accepted=False NotAllowedByListeners.

Service types and exec

Issue 3: The exec policy “didn’t work”

Testing protect-sensitive-exec, I gave a ServiceAccount RBAC to exec into OpenBao and impersonated it:

kubectl -n openbao exec --as=system:serviceaccount:kyv-test3:probe openbao-0 -- echo REACHED-SHELL
# REACHED-SHELL

The webhook was registered for CONNECT on pods/exec, yet the exec went through. The cause: my kubeconfig goes through the Tailscale operator’s API proxy in auth mode (Part 4), which impersonates my tailnet identity and drops the client’s impersonation. The request still arrived as me, a cluster admin.

Testing with the ServiceAccount’s own token, directly against the API from a controller:

TOKEN=$(k0s kubectl -n kyv-test3 create token probe --duration=10m)
k0s kubectl --kubeconfig=/dev/null --server=https://127.0.0.1:6443 \
  --certificate-authority=/opt/lib/k0s/pki/ca.crt --token=$TOKEN auth whoami
# system:serviceaccount:kyv-test3:probe
k0s kubectl ... -n openbao exec openbao-0 -- echo REACHED-SHELL
# Policy protect-sensitive-exec failed: exec/attach/port-forward into openbao is limited to cluster admins

The policy was right; the test was wrong.

Layers 3 and 4: Hygiene and Safe Defaults

• Audit (audit-workload-hygiene): images only from known registries, no :latest, a memory limit on every container. Currently it flags the GitHub runner’s :latest image and 9 namespaces without limits, so it stays Audit until those are fixed.
• Mutate (default-seccomp): pods without a seccomp profile get RuntimeDefault.
• Generate (default-deny-ingress): a namespace labeled netpol.lab/default-deny=true gets a deny-all-ingress NetworkPolicy.

All of it got one end-to-end test in a throwaway namespace:

failurePolicy stays Ignore on purpose. The next morning (Part 10), the Kyverno webhook was unreachable for a while; with Fail, every change would have been rejected.

Layer 5: Runtime Detection With Tetragon

Tetragon uses eBPF (it comes from the Cilium team, so it fits the existing stack) to see process executions, file access, syscalls and connections, with the pod and process ancestry attached. All workers run kernel 6.12 with BTF and the BPF LSM, so both monitoring and in-kernel blocking are possible. I started with monitoring only.

The export filter drops host processes, privileged system namespaces and Kubernetes health-check execs, so the output stays readable.

A simulated attacker

The first attempt never ran: the test pod asked for seccompProfile: Unconfined (so syscalls reach the probes instead of being stopped by seccomp), and layer 1 rejected it. The test ran in a namespace explicitly labeled privileged instead:

sh -c "cat /etc/shadow; cat /var/run/secrets/kubernetes.io/serviceaccount/token;
       echo x > /usr/bin/evil; unshare -n true; mount -t tmpfs none /mnt;
       wget -q -T 3 -O /dev/null http://1.1.1.1/"

exec /bin/sh
sensitive-file-access: security_file_permission ['/etc/shadow', 4] by /bin/cat
sensitive-file-access: security_file_permission ['.../serviceaccount/..2026_10_13.../token', 4] by /bin/cat
persistence-writes: security_file_permission ['/usr/bin/evil', 2] by /bin/sh
privilege-escalation: __x64_sys_unshare [1073741824] by /bin/unshare
privilege-escalation: __x64_sys_mount ['none', '/mnt'] by /bin/mount
exec /bin/wget
unexpected-egress: tcp_connect ['1.1.1.1'] by /bin/wget

Every step was caught. Normal noise from real workloads: 4 events in 10 minutes, all from Pi-hole’s internal bash.

What will (and won’t) be enforced

After a tuning period, the plan is to kill processes only for clear-cut cases: kernel module loads, ptrace, writes to /etc/shadow or /etc/ld.so.preload. Shells in openbao and databases will never be killed: my own admin workflows run kubectl exec ... sh -c there, and Tetragon can’t tell me from an attacker. Who may open those shells is Kyverno’s job (layer 2).

Lessons

1. Verify that your policy engine enforces what your config says. A rendered template showed Audit where I had asked for Deny.
2. Prefer built-ins when they exist. Pod Security Admission and Gateway API route selection needed no custom policy.
3. Test permissions with real identities. Proxies and impersonation layers can turn a correct policy into a confusing test result.
4. Detect before you enforce. Two weeks of runtime events beat guessing which processes are normal.
5. Pick your failure mode deliberately. For a homelab, availability during an incident outranks strict enforcement.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers (this post)

References

• Kubernetes: Pod Security Admission
• Kyverno: ValidatingPolicy
• Kyverno: MutatingPolicy
• Gateway API: Route attachment
• Tetragon: Tracing policies
• Tetragon: Install on Kubernetes