Homelab Rebuild Part 11: Hardening the Cluster in Five Layers

| October 13, 2026 | 10 min to read

Intro

The rebuilt cluster had Kyverno installed in audit-only mode: the Pod Security Standards as policy reports, no blocking. Time to turn that into actual protection against common misconfigurations and runtime attacks, without breaking the system components that legitimately need privileges.

TL;DR: Five layers. Pod Security Admission enforces baseline on every app namespace. Kyverno closes gaps specific to this cluster (secret scoping, Service types, exec into sensitive namespaces), audits hygiene and adds safe defaults. Tetragon watches what admission control can’t see: what processes do at runtime. Two gotchas: the Kyverno policies chart silently ignores Deny in its new CEL mode, and the Tailscale API proxy ignores kubectl --as, which made a working policy look broken.

The Figure

The grain being sieved, and the lamp of the layer that acts on it (1 to 5)Sieves 1 to 4: Pod Security Admission, cluster rules, hygiene audit, safe defaults. Pan: the running cluster, watched by TetragonAdmission control sees requests, so sieves 1 and 2 stop a privileged pod, an ExternalSecret outside its namespace path, a NodePort Service and an exec into openbao. Sieve 3 only reports a :latest image, sieve 4 injects seccompProfile RuntimeDefault into a plain pod, and a shell spawned inside a running pod reaches the pan, where Tetragon detects it without killing it. Press T to tear sieve 1: in the Kyverno chart's CEL mode, Deny was silently ignored and the privileged pod was only reported.

The five layers drawn as a sieve shaker. Each grain is a request or workload from the end-to-end test: it drops into the funnel, passes the sieves from the top, and either stops in the sieve whose layer denies it or reaches the pan, which is the running cluster. The lamp numbered after the layer lights when that layer acts, and the sensor on the post is Tetragon watching the pan. Click to drop the next grain, press 1 to 7 to pick one, and press T to tear sieve 1 the way the Kyverno chart did when it silently ignored Deny. I generated the figure with the anatomy agent skill.

The Situation

Kyverno v1.19.1, chart policies (Pod Security Standards) in Audit
Tetragon 1.7.1
System components that need privileges: Cilium and k0s’s own pods in kube-system, Tailscale proxies, local-path’s helper pods
Admission control sees API requests. It can stop a bad Pod spec; it can’t see a shell spawned inside a running container. Layer 5 covers that.

The audit reports were encouraging: of 298 would-fail results, 233 came from kube-system. My own apps were mostly clean, so enforcement could start soon, with narrow exclusions for the system namespaces.

Layer 1: Pod Security, With the Built-in Admission

Issue 1: The chart ignored Deny

My plan was to switch the chart’s baseline policies to Deny with validationFailureActionByPolicy, and exclude system namespaces with policyExclude. Rendering the chart showed neither made it into the policy:

# values.yaml
validationFailureActionByPolicy:
  disallow-host-ports: Deny
policyExclude:
  disallow-host-ports:
    any:
      - resources:
          namespaces: [kube-system, tailscale]

helm template kp kyverno-policies --repo https://kyverno.github.io/kyverno/ --version 3.9.1 -f values.yaml

# rendered disallow-host-ports: still Audit, no exclusion
validationActions:
  - Audit
matchConstraints:
  resourceRules:
    - resources: [pods]
      operations: [CREATE, UPDATE]

In the chart’s CEL mode (policyType: ValidatingPolicy), both settings are silently ignored.

Fix: Pod Security Admission

Kubernetes already ships the exact baseline standard as Pod Security Admission. One label per namespace:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest

A server-side dry run first showed whether any running pod would violate it:

kubectl label ns openbao pod-security.kubernetes.io/enforce=baseline --dry-run=server

No warnings on any of the 19 app namespaces. For namespaces created later, a Kyverno MutatingPolicy adds the label automatically (skipping kube-system, tailscale and local-path-storage). The chart’s 17 policies stay in Audit as a report of the stricter restricted level.

Layer 2: Rules Only This Cluster Needs

Issue 2: Any namespace could read every secret

External Secrets uses one cluster-wide store backed by OpenBao (Part 5). Any namespace could create an ExternalSecret pointing at any path, including the Cloudflare token. A CEL ValidatingPolicy now requires every key to sit under the namespace’s own path:

validations:
  - expression: >-
      !has(object.spec.data) ||
      object.spec.data.all(d, d.remoteRef.key.startsWith(request.namespace + "/"))
    messageExpression: >-
      "spec.data[*].remoteRef.key must start with '" + request.namespace + "/'"

It found one existing violation: external-dns read cert-manager/cloudflare. It now has its own copy at external-dns/cloudflare.

Opt-in internet exposure, natively

Apps reach the internet through a dedicated public-gateway behind a Cloudflare Tunnel (Part 9). Making that opt-in didn’t need Kyverno: Gateway API listeners can select which namespaces may attach routes.

allowedRoutes:
  namespaces:
    from: Selector
    selector:
      matchLabels:
        exposure: public

A route from an unlabeled namespace gets Accepted=False NotAllowedByListeners.

Service types and exec

Policy	Rule
`restrict-service-types`	No NodePort; LoadBalancer only in `gateway`, `pihole`, `kube-system`
`protect-sensitive-exec`	exec/attach/port-forward into `openbao`, `databases`, `cert-manager` only for `system:masters`

Issue 3: The exec policy “didn’t work”

Testing protect-sensitive-exec, I gave a ServiceAccount RBAC to exec into OpenBao and impersonated it:

kubectl -n openbao exec --as=system:serviceaccount:kyv-test3:probe openbao-0 -- echo REACHED-SHELL
# REACHED-SHELL

The webhook was registered for CONNECT on pods/exec, yet the exec went through. The cause: my kubeconfig goes through the Tailscale operator’s API proxy in auth mode (Part 4), which impersonates my tailnet identity and drops the client’s impersonation. The request still arrived as me, a cluster admin.

Testing with the ServiceAccount’s own token, directly against the API from a controller:

TOKEN=$(k0s kubectl -n kyv-test3 create token probe --duration=10m)
k0s kubectl --kubeconfig=/dev/null --server=https://127.0.0.1:6443 \
  --certificate-authority=/opt/lib/k0s/pki/ca.crt --token=$TOKEN auth whoami
# system:serviceaccount:kyv-test3:probe
k0s kubectl ... -n openbao exec openbao-0 -- echo REACHED-SHELL
# Policy protect-sensitive-exec failed: exec/attach/port-forward into openbao is limited to cluster admins

The policy was right; the test was wrong.

Layers 3 and 4: Hygiene and Safe Defaults

Audit (audit-workload-hygiene): images only from known registries, no :latest, a memory limit on every container. Currently it flags the GitHub runner’s :latest image and 9 namespaces without limits, so it stays Audit until those are fixed.
Mutate (default-seccomp): pods without a seccomp profile get RuntimeDefault.
Generate (default-deny-ingress): a namespace labeled netpol.lab/default-deny=true gets a deny-all-ingress NetworkPolicy.

All of it got one end-to-end test in a throwaway namespace:

Test	Result
New namespace	labeled `baseline` automatically
Privileged pod	denied: `violates PodSecurity "baseline:latest"`
Plain pod	`seccompProfile: RuntimeDefault` injected
NodePort / LoadBalancer in an app namespace	denied
`:latest` image	allowed, reported
Opt-in label	NetworkPolicy generated

failurePolicy stays Ignore on purpose. The next morning (Part 10), the Kyverno webhook was unreachable for a while; with Fail, every change would have been rejected.

Layer 5: Runtime Detection With Tetragon

Tetragon uses eBPF (it comes from the Cilium team, so it fits the existing stack) to see process executions, file access, syscalls and connections, with the pod and process ancestry attached. All workers run kernel 6.12 with BTF and the BPF LSM, so both monitoring and in-kernel blocking are possible. I started with monitoring only.

#	Detects	How
1	Shells, interpreters, network tools starting in a pod	export filter on exec events
2	Reads of `/etc/shadow`, SSH keys, ServiceAccount tokens by shells/tools	`security_file_permission` kprobe
3	`setuid(0)`, `ptrace`, `setns`, `unshare`, `mount`, module loads	syscall kprobes, runc excluded
4	Outbound TCP from `openbao` and `databases` outside cluster/LAN ranges	`tcp_connect` with `NotDAddr`
5	Writes to `/usr/bin`, `/etc/cron*`, `/etc/ld.so.preload`, …	`security_file_permission` with write mask

The export filter drops host processes, privileged system namespaces and Kubernetes health-check execs, so the output stays readable.

A simulated attacker

The first attempt never ran: the test pod asked for seccompProfile: Unconfined (so syscalls reach the probes instead of being stopped by seccomp), and layer 1 rejected it. The test ran in a namespace explicitly labeled privileged instead:

sh -c "cat /etc/shadow; cat /var/run/secrets/kubernetes.io/serviceaccount/token;
       echo x > /usr/bin/evil; unshare -n true; mount -t tmpfs none /mnt;
       wget -q -T 3 -O /dev/null http://1.1.1.1/"

exec /bin/sh
sensitive-file-access: security_file_permission ['/etc/shadow', 4] by /bin/cat
sensitive-file-access: security_file_permission ['.../serviceaccount/..2026_10_13.../token', 4] by /bin/cat
persistence-writes: security_file_permission ['/usr/bin/evil', 2] by /bin/sh
privilege-escalation: __x64_sys_unshare [1073741824] by /bin/unshare
privilege-escalation: __x64_sys_mount ['none', '/mnt'] by /bin/mount
exec /bin/wget
unexpected-egress: tcp_connect ['1.1.1.1'] by /bin/wget

Every step was caught. Normal noise from real workloads: 4 events in 10 minutes, all from Pi-hole’s internal bash.

What will (and won’t) be enforced

After a tuning period, the plan is to kill processes only for clear-cut cases: kernel module loads, ptrace, writes to /etc/shadow or /etc/ld.so.preload. Shells in openbao and databases will never be killed: my own admin workflows run kubectl exec ... sh -c there, and Tetragon can’t tell me from an attacker. Who may open those shells is Kyverno’s job (layer 2).

Update 2026-10-19: from watching to blocking

Most of what this post set up ran in Audit or monitor mode on purpose. After six days of data, Part 34 turned the parts with a clean record into blocks: Tetragon now kills kernel module loads, ptrace and /etc/ld.so.preload writes inside containers, protect-flux-managed and audit-workload-hygiene deny (with exits for break-glass), and kube-apiserver keeps an audit log. See Part 34.

Lessons

Verify that your policy engine enforces what your config says. A rendered template showed Audit where I had asked for Deny.
Prefer built-ins when they exist. Pod Security Admission and Gateway API route selection needed no custom policy.
Test permissions with real identities. Proxies and impersonation layers can turn a correct policy into a confusing test result.
Detect before you enforce. Two weeks of runtime events beat guessing which processes are normal.
Pick your failure mode deliberately. For a homelab, availability during an incident outranks strict enforcement.

Series

This post is part of Homelab Rebuild: