Homelab Rebuild Part 34: From Watching to Blocking, and a Record of Who Did What

| October 19, 2026 | 11 min to read

Intro

Part 11 put five layers of guardrails on the cluster, and Part 17 made them quiet enough to read. Almost all of it ran in monitor mode: Kyverno reported, Tetragon logged, nothing was stopped. That was on purpose. A guard that blocks things you didn’t expect is how you take your own cluster down.

After six days of data, three questions were left:

Which detections never fired, and can safely turn into blocks?
Which policies had no violations, and can deny?
If something does get through, is there a record of who did it?

TL;DR: Tetragon now kills three things inside containers that never happened in six days: kernel module loads, ptrace, and writes to /etc/ld.so.preload. Two Kyverno policies went from Audit to Deny, with explicit exits for break-glass. kube-apiserver now writes an audit log to VictoriaLogs. It took four tries to get right: the first version would have logged about 160,000 events a day, and it didn’t log kubectl exec at all. The final one logs about 3,000 a day.

The Figure

Approved stamp, and the lamp when Tetragon killsLedger: one line per decision (the audit log)A sends a pod with a pinned image and limits: stamped, kept. D sends one on :latest without limits, and F an operator changing a Flux-managed object: both are crossed out and slide back. B is an admin fixing a Flux object by hand (break-glass): allowed. Every decision adds a ledger line, as the kube-apiserver audit log does (about 3,000 events a day after tuning). K is a container writing /etc/ld.so.preload: no request reaches the window, the lamp lights and Tetragon kills the process (exit 137). Click to step through all five.

The figure is a bank teller’s window, because that is what admission control is. A request comes in on the pass-through tray. The teller stamps it: approved, it goes through; rejected, the tray slides it back. Every decision gets a line in the ledger, which is the audit log. The lamp on the glass is Tetragon: it lights when something already inside does what it must never do, and that process is stopped on the spot. Press A, D or B for an allowed, a denied or a break-glass request, and K for a process that gets killed. I built it with the anatomy skill (see References).

The Situation

What six days of monitor mode showed:

Guard	Detection or policy	Hits	Source
Tetragon	egress from `databases`	about 400	CNPG and Barman, all expected and allowed since
Tetragon	`/etc/shadow` read	80	`sudo` in the CI runners
Tetragon	writes to `/usr/bin` and `/etc/cron`	45	`install` in the CI runners, Pi-hole’s `crontab`
Tetragon	`setuid(0)`	9	Pi-hole’s own cron
Tetragon	kernel module load	0
Tetragon	`ptrace`	0
Tetragon	write to `/etc/ld.so.preload`	0
Kyverno	`protect-flux-managed`	0 in 7 days
Kyverno	`audit-workload-hygiene`	28 report failures, 212 admission events	old ReplicaSets, and the CI runner’s `:latest` image

The rule for turning a guard into a block: it must have a clean record, and everything it would have stopped must be something I never want.

Tetragon: Kill Only What Never Fires

The three detections with zero hits are also the three with no innocent explanation inside a container. Loading a kernel module, attaching to another process with ptrace, or adding a library to /etc/ld.so.preload (which every program on the system then loads) are what escapes and rootkits do, not what apps do.

A new TracingPolicy sends Sigkill for exactly those:

- call: security_file_permission
  syscall: false
  args:
    - index: 0
      type: file
    - index: 1
      type: int
  selectors:
    - matchNamespaces:
        - namespace: Pid
          operator: NotIn
          values: [host_ns]
      matchArgs:
        - index: 0
          operator: Equal
          values: [/etc/ld.so.preload]
        - index: 1
          operator: Mask
          values: ["2"]   # MAY_WRITE
      matchActions:
        - action: Sigkill

The matchNamespaces selector is the important part. The existing policies are cluster-wide, so they also see processes on the VMs themselves, where a module load is a normal part of boot. Pid NotIn host_ns limits the kill to processes in a container’s PID namespace.

Everything else stays monitor-only. /etc/shadow reads would kill the CI runners’ sudo. Shells are never killed in openbao or databases, because my own admin scripts run there, and Kyverno already limits who can open them.

The test was a throwaway pod:

command: sh -c "echo before; echo x > /etc/ld.so.preload; echo survived"
phase=Failed exit=137
before

survived never printed, and the event arrived in VictoriaLogs with "action":"KPROBE_ACTION_SIGKILL", so the existing Tetragon alert reports every kill.

Kyverno: Deny, With Exits

Flux-managed objects

protect-flux-managed stops anyone but Flux from changing an object Flux applied from git. Seven days in Audit had zero violations, so it could deny. But a deny with no way around it would have broken two things:

Break-glass. My documented emergency path is “suspend the Flux Kustomization, then fix by hand”. Cluster admins (system:masters) are now exempt.
Namespace deletion. When an app is removed, Kubernetes’ namespace controller and garbage collector delete what’s inside. Without an exception they would be denied, and the namespace would hang in Terminating.

- expression: >-
    request.userInfo.username in variables.fluxControllers ||
    "system:masters" in request.userInfo.groups ||
    variables.annotationOnly

fluxControllers now includes the two kube-system collectors, and annotationOnly still lets flux reconcile through.

Workload hygiene

audit-workload-hygiene requires trusted registries, pinned image tags, a memory limit and requests. Its reports looked bad: 28 failures. Every one was an old ReplicaSet kept for rollback, from before limits existed, and every live pod passed.

Reports only show what exists right now, though. Short-lived pods come and go between scans, so the admission events were the real evidence: 212 failures, all from CI runner pods, all for one reason:

images must be pinned to a tag other than :latest, or a digest

The runner image was ghcr.io/actions/actions-runner:latest. Pinning it to 2.338.0 was the whole fix. Version bumps are now a pull request, which they should have been anyway.

Testing the denies

$ kubectl run hygiene-test --image=busybox:latest --dry-run=server
denied the request: Policy audit-workload-hygiene failed: images must be pinned to a tag other than :latest, or a digest

$ kubectl label cluster pg kyverno-test=1 --dry-run=server --as=system:serviceaccount:cnpg-system:cnpg-cloudnative-pg
denied the request: Policy protect-flux-managed failed: Flux-managed object changed outside git by system:serviceaccount:cnpg-system:cnpg-cloudnative-pg

$ kubectl label cluster pg kyverno-test=1 --dry-run=server     # as admin
cluster.postgresql.cnpg.io/pg labeled (server dry run)

The impersonation test had to run on a controller: the Tailscale API proxy ignores --as, as Part 11 found. Kyverno still fails open if it is down (forceFailurePolicyIgnore), so an outage at the 06:00 boot can never block the API.

The Audit Log

kube-apiserver can write an audit event for every request. The controllers' journals already go to VictoriaLogs (the Part 28 update), so the simplest path was to write the audit log to stdout:

# k0sctl.yaml, spec.api.extraArgs
audit-policy-file: /etc/k0s/audit-policy.yaml
audit-log-path: "-"

A flag change means restarting kube-apiserver on all three controllers, one at a time, checking etcd, the controller leases and the konnectivity count in between. Each controller answered again after about six seconds. The apiserver reads the policy file only at startup, so every policy change was another round of three restarts. There were four rounds.

Issue 1: About 160,000 events a day

The first policy logged every write and dropped every read. One controller logged 1,357 events in 12 minutes:

422  kyverno-reports-controller  update  policyreports
348  keda-operator               update  apiservices
348  keda-operator               update  validatingwebhookconfigurations

Kyverno rewrites its own report objects constantly, and KEDA re-writes the certificate data on its APIService and webhook every two seconds. Neither is news. Both are now excluded, but only for those identities: anyone else changing a webhook is still logged.

Issue 2: kubectl exec was not in the log

A test kubectl exec didn’t show up anywhere. Current kubectl runs exec over WebSockets, and the upgrade request arrives as a get on pods/exec. “Drop all reads” had dropped exactly the request I most wanted to see. The fix is a rule at the top:

- level: Metadata
  resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]

Issue 3: I was a ServiceAccount

The same test showed who the log thought I was:

"user":{"username":"system:serviceaccount:tailscale:operator", ...}

Everything I do through the tailnet kubeconfig reaches the apiserver as the Tailscale proxy’s own ServiceAccount. That had just mattered, because I had also told the policy to skip Secret reads by ServiceAccounts (operators read Secrets all day). My own Secret reads would have vanished. The proxy’s account now gets its own rule before the ServiceAccount exclusion. The log can’t tell me apart from anything else using the proxy, but it does record that the proxy did it.

Issue 4: Flux applying git

With those fixed, Flux was most of what was left: every reconcile server-side-applies each object, including a dry run for drift detection, about 65,000 patches a day. Git already records those changes, so Flux’s create, update and patch are excluded, and its deletes stay. Status updates from controllers went the same way.

The final policy

Logged (Metadata only, never bodies)	Not logged
exec, attach, port-forward, any verb	reads other than Secrets
writes by people and operators	Flux applying git (its deletes stay)
Secret reads by people, and by the Tailscale proxy	controller status updates
deletes by Flux	Kyverno reports, KEDA certificate refresh, events, leases

Ten minutes after the last round: 21 events across the three controllers, about 3,000 a day. The first useful answer came minutes later. I wanted proof that CI runner pods were still admitted after the hygiene Deny, and the runner pod events in VictoriaLogs were ambiguous. The audit log was not:

k0s-controller-03 02:19:19 create homelab-runner-t5p44 201
k0s-controller-02 02:21:09 create homelab-runner-2svjc 201

Results

Guard	Before	After
Tetragon	5 policies, monitor only	the same, plus Sigkill on module loads, `ptrace` and `/etc/ld.so.preload` writes in containers
Kyverno `protect-flux-managed`	Audit	Deny, except Flux, admins and the collectors
Kyverno `audit-workload-hygiene`	Audit, runner on `:latest`	Deny, runner pinned
kube-apiserver audit	none	about 3,000 events a day in VictoriaLogs
Controller restarts		12, one at a time, about 6 seconds each

Lessons

Enforce what has a clean record, and nothing else. Six days of zero hits made three kills safe; 80 /etc/shadow reads by sudo kept that one in monitor mode.
Scope runtime kills to containers. A cluster-wide kprobe also sees the hosts, where module loads are normal.
Every Deny needs its exits written down. Break-glass and the namespace controller are both legitimate “outside git” changes.
Policy reports show what exists, admission events show what happened. The reports blamed old ReplicaSets; the events found the CI runner.
Test an audit policy against the requests you care about. Mine dropped kubectl exec because the client changed how it sends it.
Know who the apiserver thinks you are. Behind a proxy, it may not be you.
Measure before deciding what to drop. Two machine loops were most of the first version’s volume, and both were easy to name once counted.

Series

This post is part of Homelab Rebuild: