Intro
The rebuilt cluster had Kyverno installed in audit-only mode: the Pod Security Standards as policy reports, with no blocking. I wanted to turn that into real protection against common misconfigurations and runtime attacks. I also did not want to break the system components that legitimately need privileges.
TL;DR: Five layers. Pod Security Admission enforces
baselineon every app namespace. Kyverno closes gaps specific to this cluster (secret scoping, Service types, exec into sensitive namespaces). It also audits hygiene and adds safe defaults. Tetragon watches what admission control cannot see: what processes do at runtime. Two gotchas: the Kyverno policies chart ignoresDenywithout a warning in its new CEL mode. The Tailscale API proxy ignoreskubectl --as, so a working policy looked broken.
The Figure
The figure draws the five layers as a sieve shaker. Each grain is a request or
workload from the end-to-end test. It drops into the funnel and passes the sieves
from the top. It either stops in the sieve whose layer denies it, or it reaches
the pan, which is the running cluster. The lamp with the number of the layer
lights when that layer acts. The sensor on the post is Tetragon, which watches
the pan. Click to drop the next grain, and press 1 to 7 to pick one. Press
T to tear sieve 1, the way the Kyverno chart did when it ignored Deny. I generated the
figure with the anatomy agent skill.
The Situation
- Kyverno v1.19.1, chart policies (Pod Security Standards) in Audit
- Tetragon 1.7.1
- System components that need privileges: Cilium and the k0s pods in
kube-system, Tailscale proxies, the local-path helper pods - Admission control sees API requests. It can stop a bad Pod spec. It cannot see a shell spawned inside a running container. Layer 5 covers that.
The audit reports were encouraging. Of 298 would-fail results, 233 came from
kube-system. My own apps were mostly clean. So I planned to start enforcement soon,
with narrow exclusions for the system namespaces.
Layer 1: Pod Security, With the Built-in Admission
Issue 1: The chart ignored Deny
My plan was to switch the baseline policies of the chart to Deny with
validationFailureActionByPolicy. I also wanted to exclude system namespaces
with policyExclude. When I rendered the chart, neither setting was in the
policy:
# values.yaml
validationFailureActionByPolicy:
disallow-host-ports: Deny
policyExclude:
disallow-host-ports:
any:
- resources:
namespaces: [kube-system, tailscale]
helm template kp kyverno-policies --repo https://kyverno.github.io/kyverno/ --version 3.9.1 -f values.yaml
# rendered disallow-host-ports: still Audit, no exclusion
validationActions:
- Audit
matchConstraints:
resourceRules:
- resources: [pods]
operations: [CREATE, UPDATE]
In the CEL mode of the chart (policyType: ValidatingPolicy), the chart ignores
both settings and gives no warning.
Fix: Pod Security Admission
Kubernetes already ships the exact baseline standard as Pod Security Admission. One label per namespace:
metadata:
labels:
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/enforce-version: latest
First, a server-side dry run showed if any running pod violates it:
kubectl label ns openbao pod-security.kubernetes.io/enforce=baseline --dry-run=server
I got no warnings on any of the 19 app namespaces. For namespaces created later,
a Kyverno MutatingPolicy adds the label automatically. It skips kube-system,
tailscale and local-path-storage. The 17 policies of the chart stay in Audit
as a report of the stricter restricted level.
Layer 2: Rules Only This Cluster Needs
Issue 2: Any namespace could read every secret
External Secrets uses one cluster-wide store backed by OpenBao (Part 5). Any
namespace was able to create an ExternalSecret that points at any path, including the
Cloudflare token. A CEL ValidatingPolicy now requires every key to sit under
the path of its own namespace:
validations:
- expression: >-
!has(object.spec.data) ||
object.spec.data.all(d, d.remoteRef.key.startsWith(request.namespace + "/"))
messageExpression: >-
"spec.data[*].remoteRef.key must start with '" + request.namespace + "/'"
It found one existing violation: external-dns read cert-manager/cloudflare.
It now has its own copy at external-dns/cloudflare.
Opt-in internet exposure, natively
Apps reach the internet through a dedicated public-gateway behind a
Cloudflare Tunnel (Part 9). To make that opt-in, I did not need Kyverno. Gateway API
listeners can select which namespaces can attach routes.
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
exposure: public
A route from an unlabeled namespace gets Accepted=False NotAllowedByListeners.
Service types and exec
| Policy | Rule |
|---|---|
restrict-service-types | No NodePort; LoadBalancer only in gateway, pihole, kube-system |
protect-sensitive-exec | exec/attach/port-forward into openbao, databases, cert-manager only for system:masters |
Issue 3: The exec policy “did not work”
To test protect-sensitive-exec, I gave a ServiceAccount RBAC to exec into
OpenBao and impersonated it:
kubectl -n openbao exec --as=system:serviceaccount:kyv-test3:probe openbao-0 -- echo REACHED-SHELL
# REACHED-SHELL
The webhook was registered for CONNECT on pods/exec, but the exec went
through. The cause: my kubeconfig goes through the API proxy of the Tailscale
operator in auth mode (Part 4). The proxy impersonates my tailnet identity and
drops the impersonation from the client. The request still arrived as me, a
cluster admin.
Then I tested with the token of the ServiceAccount, directly against the API from a controller:
TOKEN=$(k0s kubectl -n kyv-test3 create token probe --duration=10m)
k0s kubectl --kubeconfig=/dev/null --server=https://127.0.0.1:6443 \
--certificate-authority=/opt/lib/k0s/pki/ca.crt --token=$TOKEN auth whoami
# system:serviceaccount:kyv-test3:probe
k0s kubectl ... -n openbao exec openbao-0 -- echo REACHED-SHELL
# Policy protect-sensitive-exec failed: exec/attach/port-forward into openbao is limited to cluster admins
The policy was right. The test was wrong.
Layers 3 and 4: Hygiene and Safe Defaults
- Audit (
audit-workload-hygiene): images only from known registries, no:latest, a memory limit on every container. Currently it flags the:latestimage of the GitHub runner and 9 namespaces without limits. It stays in Audit until I fix those. - Mutate (
default-seccomp): pods without a seccomp profile getRuntimeDefault. - Generate (
default-deny-ingress): a namespace labelednetpol.lab/default-deny=truegets a deny-all-ingress NetworkPolicy.
I tested all of it once, end to end, in a throwaway namespace:
| Test | Result |
|---|---|
| New namespace | labeled baseline automatically |
| Privileged pod | denied: violates PodSecurity "baseline:latest" |
| Plain pod | seccompProfile: RuntimeDefault injected |
| NodePort / LoadBalancer in an app namespace | denied |
:latest image | allowed, reported |
| Opt-in label | NetworkPolicy generated |
failurePolicy stays Ignore on purpose. The next morning (Part 10), the
Kyverno webhook was unreachable for a while. With Fail, the cluster rejects
every change during that time.
Layer 5: Runtime Detection With Tetragon
Tetragon uses eBPF (programs that run inside the kernel) to see process executions, file access, syscalls and connections. Each event has the pod and the process ancestry attached. Tetragon comes from the Cilium team, so it fits the existing stack. All workers run kernel 6.12 with BTF and the BPF LSM. So both monitoring and in-kernel blocking are possible. I started with monitoring only.
| # | Detects | How |
|---|---|---|
| 1 | Shells, interpreters, network tools starting in a pod | export filter on exec events |
| 2 | Reads of /etc/shadow, SSH keys, ServiceAccount tokens by shells/tools | security_file_permission kprobe |
| 3 | setuid(0), ptrace, setns, unshare, mount, module loads | syscall kprobes, runc excluded |
| 4 | Outbound TCP from openbao and databases outside cluster/LAN ranges | tcp_connect with NotDAddr |
| 5 | Writes to /usr/bin, /etc/cron*, /etc/ld.so.preload, … | security_file_permission with write mask |
The export filter drops host processes, privileged system namespaces and Kubernetes health-check execs, so the output stays readable.
A simulated attacker
The first attempt never ran. The test pod asked for
seccompProfile: Unconfined, so that syscalls reach the probes and seccomp does
not stop them. Layer 1 rejected it. So I ran the test in a namespace explicitly
labeled privileged:
sh -c "cat /etc/shadow; cat /var/run/secrets/kubernetes.io/serviceaccount/token;
echo x > /usr/bin/evil; unshare -n true; mount -t tmpfs none /mnt;
wget -q -T 3 -O /dev/null http://1.1.1.1/"
exec /bin/sh
sensitive-file-access: security_file_permission ['/etc/shadow', 4] by /bin/cat
sensitive-file-access: security_file_permission ['.../serviceaccount/..2026_10_13.../token', 4] by /bin/cat
persistence-writes: security_file_permission ['/usr/bin/evil', 2] by /bin/sh
privilege-escalation: __x64_sys_unshare [1073741824] by /bin/unshare
privilege-escalation: __x64_sys_mount ['none', '/mnt'] by /bin/mount
exec /bin/wget
unexpected-egress: tcp_connect ['1.1.1.1'] by /bin/wget
Tetragon caught every step. Normal noise from real workloads was 4 events in 10
minutes, all from the internal bash of Pi-hole.
What will (and will not) be enforced
After a tuning period, I plan to kill processes only for clear-cut cases:
kernel module loads, ptrace, writes to /etc/shadow or /etc/ld.so.preload.
Tetragon will never kill shells in openbao and databases. My own admin
workflows run kubectl exec ... sh -c there, and Tetragon cannot tell me from an
attacker. Kyverno decides who can open those shells (layer 2).
Update 2026-10-19: from watching to blocking
Most of what this post set up ran in Audit or monitor mode on purpose. After
six days of data, Part 34 turned the parts with a clean record into blocks.
Tetragon now kills kernel module loads, ptrace and /etc/ld.so.preload
writes inside containers. protect-flux-managed and audit-workload-hygiene
deny (with exits for break-glass, an emergency override). kube-apiserver keeps
an audit log. See
Part 34.
Lessons
- Verify that your policy engine enforces what your config says. A
rendered template showed
Auditwhere I asked forDeny. - Prefer built-ins when they exist. Pod Security Admission and Gateway API route selection needed no custom policy.
- Test permissions with real identities. Proxies and impersonation layers can turn a correct policy into a confusing test result.
- Detect before you enforce. Two weeks of runtime events beat guessing which processes are normal.
- Pick your failure mode deliberately. For a homelab, availability during an incident outranks strict enforcement.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers (this post)
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
