Homelab Rebuild Part 34: From Watching to Blocking, and a Record of Who Did What

| October 19, 2026 | 11 min to read

Intro

Part 11 put five layers of guardrails on the cluster, and Part 17 made them quiet enough to read. Almost all of it ran in monitor mode (report only, block nothing). Kyverno reported, Tetragon logged, and nothing was stopped. That was on purpose. A guard that blocks things you did not expect is how you take your own cluster down.

After six days of data, three questions were left:

Which detections never fired, and can turn into blocks without risk?
Which policies had no violations, and can deny?
If something does get through, is there a record of who did it?

TL;DR: Tetragon now kills three things inside containers that never happened in six days: kernel module loads, ptrace, and writes to /etc/ld.so.preload. Two Kyverno policies went from Audit to Deny, with explicit exits for break-glass (emergency access). kube-apiserver now writes an audit log to VictoriaLogs. It took four tries to get right. The first version was on track to log about 160,000 events a day, and it did not log kubectl exec at all. The final one logs about 3,000 a day.

The Figure

Approved stamp, and the lamp when Tetragon killsLedger: one line per decision (the audit log)A sends a pod with a pinned image and limits: stamped, kept. D sends one on :latest without limits, and F an operator changing a Flux-managed object: both are crossed out and slide back. B is an admin fixing a Flux object by hand (break-glass): allowed. Every decision adds a ledger line, as the kube-apiserver audit log does (about 3,000 events a day after tuning). K is a container writing /etc/ld.so.preload: no request reaches the window, the lamp lights and Tetragon kills the process (exit 137). Click to step through all five.

The figure is a bank teller’s window, because that is what admission control is. A request comes in on the pass-through tray, and the teller stamps it. If approved, it goes through. If rejected, the tray slides it back. Every decision gets a line in the ledger, which is the audit log. The lamp on the glass is Tetragon. It lights when something already inside does what it must never do, and that process is stopped on the spot. Press A, D or B for an allowed, a denied or a break-glass request. Press K for a process that gets killed. I built it with the anatomy skill (see References).

The Situation

What six days of monitor mode showed:

Guard	Detection or policy	Hits	Source
Tetragon	egress from `databases`	about 400	CNPG and Barman, all expected and allowed since
Tetragon	`/etc/shadow` read	80	`sudo` in the CI runners
Tetragon	writes to `/usr/bin` and `/etc/cron`	45	`install` in the CI runners, Pi-hole’s `crontab`
Tetragon	`setuid(0)`	9	Pi-hole’s own cron
Tetragon	kernel module load	0
Tetragon	`ptrace`	0
Tetragon	write to `/etc/ld.so.preload`	0
Kyverno	`protect-flux-managed`	0 in 7 days
Kyverno	`audit-workload-hygiene`	28 report failures, 212 admission events	old ReplicaSets, and the CI runner’s `:latest` image

The rule for turning a guard into a block: it must have a clean record. Also, everything that it stops must be something I never want.

Tetragon: Kill Only What Never Fires

The three detections with zero hits are also the three with no innocent explanation inside a container. Escapes and rootkits do these three things. They load a kernel module, attach to another process with ptrace, or add a library to /etc/ld.so.preload (every program on the system then loads it). No app of mine needs any of them.

A new TracingPolicy sends Sigkill for exactly those:

- call: security_file_permission
  syscall: false
  args:
    - index: 0
      type: file
    - index: 1
      type: int
  selectors:
    - matchNamespaces:
        - namespace: Pid
          operator: NotIn
          values: [host_ns]
      matchArgs:
        - index: 0
          operator: Equal
          values: [/etc/ld.so.preload]
        - index: 1
          operator: Mask
          values: ["2"]   # MAY_WRITE
      matchActions:
        - action: Sigkill

The matchNamespaces selector is the important part. The existing policies are cluster-wide, so they also see processes on the VMs themselves, where a module load is a normal part of boot. Pid NotIn host_ns limits the kill to processes in a container’s PID namespace.

Everything else stays monitor-only. A kill on /etc/shadow reads also kills the sudo of the CI runners. Tetragon never kills shells in openbao or databases, because my own admin scripts run there. Kyverno already limits who can open them.

The test was a throwaway pod:

command: sh -c "echo before; echo x > /etc/ld.so.preload; echo survived"
phase=Failed exit=137
before

survived never printed. The event arrived in VictoriaLogs with "action":"KPROBE_ACTION_SIGKILL", so the existing Tetragon alert reports every kill.

Kyverno: Deny, With Exits

Flux-managed objects

protect-flux-managed stops anyone but Flux from changing an object Flux applied from git. Seven days in Audit had zero violations, so it was ready to deny. But a deny with no way around it breaks two things:

Break-glass. My documented emergency path is “suspend the Flux Kustomization, then fix by hand”. Cluster admins (system:masters) are now exempt.
Namespace deletion. When an app is removed, Kubernetes’ namespace controller and garbage collector delete what is inside. Without an exception, the policy denies them, and the namespace hangs in Terminating.

- expression: >-
    request.userInfo.username in variables.fluxControllers ||
    "system:masters" in request.userInfo.groups ||
    variables.annotationOnly

fluxControllers now includes the two kube-system collectors, and annotationOnly still lets flux reconcile through.

Workload hygiene

audit-workload-hygiene requires trusted registries, pinned image tags, a memory limit and requests. Its reports looked bad: 28 failures. Every one was an old ReplicaSet kept for rollback, from before limits existed. Every live pod passed.

But reports only show what exists right now. Short-lived pods come and go between scans, so the admission events were the real evidence. They showed 212 failures, all from CI runner pods, all for one reason:

images must be pinned to a tag other than :latest, or a digest

The runner image was ghcr.io/actions/actions-runner:latest. Pinning it to 2.338.0 was the whole fix. Version bumps are now a pull request, and that is how it always had to be.

Testing the denies

$ kubectl run hygiene-test --image=busybox:latest --dry-run=server
denied the request: Policy audit-workload-hygiene failed: images must be pinned to a tag other than :latest, or a digest

$ kubectl label cluster pg kyverno-test=1 --dry-run=server --as=system:serviceaccount:cnpg-system:cnpg-cloudnative-pg
denied the request: Policy protect-flux-managed failed: Flux-managed object changed outside git by system:serviceaccount:cnpg-system:cnpg-cloudnative-pg

$ kubectl label cluster pg kyverno-test=1 --dry-run=server     # as admin
cluster.postgresql.cnpg.io/pg labeled (server dry run)

The impersonation test had to run on a controller. The Tailscale API proxy ignores --as, as Part 11 found. Kyverno still fails open if it is down (forceFailurePolicyIgnore). So an outage at the 06:00 boot can never block the API.

The Audit Log

kube-apiserver can write an audit event for every request. The journals of the controllers already go to VictoriaLogs (the Part 28 update). So the simplest path was to write the audit log to stdout:

# k0sctl.yaml, spec.api.extraArgs
audit-policy-file: /etc/k0s/audit-policy.yaml
audit-log-path: "-"

A flag change means a restart of kube-apiserver on all three controllers, one at a time. Between restarts, I checked etcd, the controller leases and the konnectivity count. Each controller answered again after about six seconds. The apiserver reads the policy file only at startup. So every policy change was another round of three restarts. There were four rounds.

Issue 1: About 160,000 events a day

The first policy logged every write and dropped every read. One controller logged 1,357 events in 12 minutes:

422  kyverno-reports-controller  update  policyreports
348  keda-operator               update  apiservices
348  keda-operator               update  validatingwebhookconfigurations

Kyverno rewrites its own report objects all the time. KEDA rewrites the certificate data on its APIService and webhook every two seconds. Neither is news. Both are now excluded, but only for those identities. If anyone else changes a webhook, the log still records it.

Issue 2: kubectl exec was not in the log

A test kubectl exec did not show up anywhere. Current kubectl runs exec over WebSockets, and the upgrade request arrives as a get on pods/exec. “Drop all reads” dropped exactly the request that I most wanted to see. The fix is a rule at the top:

- level: Metadata
  resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]

Issue 3: I was a ServiceAccount

The same test showed who the log thought I was:

"user":{"username":"system:serviceaccount:tailscale:operator", ...}

Everything I do through the tailnet kubeconfig reaches the apiserver as the Tailscale proxy’s own ServiceAccount. That mattered right away. I also told the policy to skip Secret reads by ServiceAccounts (operators read Secrets all day). That rule alone drops my own Secret reads from the log. The proxy’s account now gets its own rule before the ServiceAccount exclusion. The log cannot tell me apart from anything else that uses the proxy, but it does record that the proxy did it.

Issue 4: Flux applying git

With those fixed, Flux was most of what was left. Every reconcile server-side-applies each object, including a dry run for drift detection. That is about 65,000 patches a day. Git already records those changes. So the policy excludes Flux’s create, update and patch, and its deletes stay. Status updates from controllers went the same way.

The final policy

Logged (Metadata only, never bodies)	Not logged
exec, attach, port-forward, any verb	reads other than Secrets
writes by people and operators	Flux applying git (its deletes stay)
Secret reads by people, and by the Tailscale proxy	controller status updates
deletes by Flux	Kyverno reports, KEDA certificate refresh, events, leases

Ten minutes after the last round: 21 events across the three controllers, about 3,000 a day. The first useful answer came minutes later. I wanted proof that CI runner pods were still admitted after the hygiene Deny. The runner pod events in VictoriaLogs were ambiguous. The audit log was not:

k0s-controller-03 02:19:19 create homelab-runner-t5p44 201
k0s-controller-02 02:21:09 create homelab-runner-2svjc 201

Results

Guard	Before	After
Tetragon	5 policies, monitor only	the same, plus Sigkill on module loads, `ptrace` and `/etc/ld.so.preload` writes in containers
Kyverno `protect-flux-managed`	Audit	Deny, except Flux, admins and the collectors
Kyverno `audit-workload-hygiene`	Audit, runner on `:latest`	Deny, runner pinned
kube-apiserver audit	none	about 3,000 events a day in VictoriaLogs
Controller restarts		12, one at a time, about 6 seconds each

Lessons

Enforce what has a clean record, and nothing else. Six days of zero hits made three kills safe. 80 /etc/shadow reads by sudo kept that one in monitor mode.
Scope runtime kills to containers. A cluster-wide kprobe also sees the hosts, where module loads are normal.
Every Deny needs its exits written down. Break-glass and the namespace controller are both legitimate “outside git” changes.
Policy reports show what exists, admission events show what happened. The reports blamed old ReplicaSets. The events found the CI runner.
Test an audit policy against the requests you care about. Mine dropped kubectl exec because the client changed how it sends it.
Know who the apiserver thinks you are. Behind a proxy, the answer can be someone else.
Measure before deciding what to drop. Two machine loops were most of the first version’s volume. Both were easy to name after I counted them.

Series

This post is part of Homelab Rebuild: