Homelab Rebuild Part 17: Making the Guardrails Earn Their Keep

Intro

With Flux in charge (Part 16), I went through the cluster and asked one question about every component: does anyone use this? Several answers were no. Kyverno had 21 policies, all in Audit, with 566 failures that nobody read. Tetragon had five detection policies that wrote events nobody looked at. KEDA had zero ScaledObjects. The CloudNativePG cluster had zero client connections.

This post covers how I made the first two useful and added a Flux view to Headlamp. It also covers the first job for Postgres.

TL;DR: Most Kyverno failures were system pods that need privileges by design. Two PolicyExceptions took the failures from 566 to 246. The rest is a real fix list. Three new rules target mistakes I actually made. The first refuses anything in default. The top two noise sources in Tetragon were IPv6 loopback (::1) and the restore_command of Postgres itself. After tuning, it exported zero events in 30 minutes. A Grafana alert on VictoriaLogs now watches for policy hits. Postgres got a homelab database for queues, pub/sub, key/value and cache, explicitly best effort with no SLA.

The Situation

• k0s cluster on three Proxmox hosts. Flux reconciles 20 apps from Git
• Kyverno 1.19.1 (chart 3.9.1): the 17 Pod Security Standards (PSS) policies of the chart plus my own from Part 11. All are ValidatingPolicy (CEL), mostly Audit
• Tetragon 1.7.1: five policies in monitor mode. Events go to stdout, and the OTel agent collects them into VictoriaLogs
• CloudNativePG 1.30.1, cluster pg with two instances, no backups yet
• Headlamp 0.45.0 as the cluster UI
• Decisions at the start: keep KEDA for future queue workers, and skip backups until I pick an object store. Also, use Postgres for everything instead of adding Redis or a message broker

Issue 1: 566 Kyverno failures, almost all expected

Symptom

kubectl get policyreports -A -o json | jq -r '.items[].results[]? | select(.result=="fail") | .policy' | sort | uniq -c | sort -rn | head -5
# 43 restrict-seccomp-strict
# 40 disallow-host-namespaces
# 29 require-run-as-nonroot
# 28 disallow-host-ports
# 24 disallow-capabilities-strict

There were 283 reports and 566 failures. The top entries were all in kube-system: Cilium, Tetragon and the k0s components. They need host networking, host paths and capabilities to do their job. With that much expected red, a real problem in an app namespace never stands out.

Cause

The kyverno-policies chart has policyExclude for exactly this. But in CEL mode (ValidatingPolicy), the chart ignores it. Exclusions must be PolicyException objects, and that feature was off. A server-side dry-run of my first exception said so:

Warning: PolicyException resources would not be processed until it is enabled.

Fix

Enable exceptions, but only from the kyverno namespace. Then an app namespace cannot exempt itself:

features:
  policyExceptions:
    enabled: true
    namespace: kyverno

Then add two exceptions. One exempts whole system namespaces from the 17 PSS policies. The other exempts only the OTel agent DaemonSet, which reads host logs and metrics:

apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: system-namespaces
  namespace: kyverno
spec:
  policyRefs:
    - {kind: ValidatingPolicy, name: disallow-host-namespaces}
    - {kind: ValidatingPolicy, name: disallow-host-path}
    # ... the other 15 PSS policies
  matchConditions:
    - name: system-namespace
      expression: >-
        object.metadata.namespace in ["kube-system", "tailscale", "local-path-storage"]

Everything else stays in the reports on purpose. Grafana, vmsingle, the ARC controller, Pi-hole and others now make up a fix list, not a list of exceptions.

The Helm upgrade itself had a hiccup. Release v5 stayed in pending-upgrade. Flux unlocked it, and v6 succeeded. The reports then took a while to catch up:

At that point, 66 failures were still listed in exempt namespaces. All had timestamps from before the exceptions existed. They are old results that wait for the next background scan, not new failures.

Issue 2: New rules, and a policy that stayed not ready

I added rules for mistakes that I actually made:

The deny works:

kubectl run deny-test --image=busybox:1.37 -n default --dry-run=server
# Error from server: admission webhook "vpol.validate.kyverno.svc-ignore-9d71e4df" denied the request:
# Policy deny-default-namespace failed: nothing runs in the default namespace: set metadata.namespace (or use the app's kustomization)

Symptom

deny-default-namespace showed READY false, but it already blocked requests:

RBACPermissionsGranted False Failed Policy is not ready for reporting, missing permissions:
get postgresql.cnpg.io/v1, Resource=clusters: ; list ...; watch ...

Cause

The policy also matches CNPG Cluster objects. The Kyverno reports controller had no permission to read them. Enforcement needs only the admission webhook, so blocking still worked. Only the reporting was missing.

Fix

Add a read-only ClusterRole. The Kyverno aggregation label merges it into the reports controller:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:reports-controller:cnpg
  labels:
    rbac.kyverno.io/aggregate-to-reports-controller: "true"
rules:
  - apiGroups: ["postgresql.cnpg.io"]
    resources: ["clusters"]
    verbs: ["get", "list", "watch"]

The role was aggregated correctly, but the policy stayed READY false. The timestamp of the condition never moved. Kyverno rechecks permissions only when the policy changes. So I restarted the admission and reports controllers. After that, all 24 policies were ready.

A rule that flagged me first

protect-flux-managed matches every resource type, and that worried me. If Kyverno does not pass the label filter through to its webhook, every update in the cluster goes through Kyverno. It did pass it through:

kyverno-resource-validating-webhook-cfg vpol.validate.kyverno.svc-ignore-7e7ae6df
  {"matchExpressions": [{"key": "kustomize.toolkit.fluxcd.io/name", "operator": "Exists"}]}

Its first report was about me. I set the reconcile.fluxcd.io/requestedAt annotation to make Flux reconcile immediately. flux reconcile and the Headlamp sync button do the same. Updates that leave the spec and labels untouched now pass:

- name: annotationOnly
  expression: >-
    request.operation == "UPDATE" &&
    object.?spec == oldObject.?spec &&
    object.metadata.?labels == oldObject.metadata.?labels

A suspend of an app from Headlamp changes spec.suspend, and Kyverno still reports it. That is the point: it is drift from Git.

Issue 3: Tetragon was mostly watching Postgres

Symptom

Tetragon events reached VictoriaLogs, so I counted them by type, policy and binary. This is about 13 hours of data:

321 process_kprobe  unexpected-egress  databases  /controller/manager
295 process_exec    databases          /bin/sh
 24 process_exec    default            /bin/sh
  5 process_kprobe  persistence-writes pihole     /usr/bin/crontab

The 24 shells in default were the duplicate Postgres cluster from Part 15. The rest needed a closer look.

Cause

The egress policy reports outbound connections from databases and openbao to anything outside the cluster, LAN and loopback ranges. 315 of the 321 events went to the status port of the CNPG instance manager itself, over IPv6:

{"d":"::1","port":"8010","n":"315"}

The allow list had 127.0.0.0/8 but not ::1.

The 295 shells came from standby pg-1. Their parent was postgres, and the command was the CNPG restore_command:

sh -c -- "/controller/manager wal-restore --log-destination /controller/log/postgres.json 00000005.history pg_wal/RECOVERYHISTORY"

A standby runs it for every WAL file and timeline that it looks for. All 295 ran between 02:41 and 12:11 WIB. That window had two outages and their recoveries. None ran in the two hours before I looked.

The five crontab writes were Pi-hole, which manages its own schedule.

Fix

Add ::1/128 to both egress allow lists. Then drop the two known-benign repeats at export. Match them as narrowly as the Tetragon filters allow:

exportDenyList: |-
  {"health_check":true}
  {"namespace":["","kube-system","tailscale","local-path-storage"]}
  {"namespace":["databases"],"binary_regex":["^/bin/sh$"],"arguments_regex":["/controller/manager wal-restore "]}
  {"namespace":["pihole"],"binary_regex":["^/usr/bin/crontab$"],"policy_names":["persistence-writes"]}

My first version had a # comment line inside that block. The block is JSON lines, not YAML, so the comment breaks the filter parsing. I caught it before the commit and moved the comment above the key.

I also added a bpf() hook to the privilege escalation policy. A pod that loads eBPF programs uses a classic escape primitive (a basic building block for a container escape). Cilium and Tetragon load them legitimately, so I excluded them. Right after the rollout, tetra tracingpolicy list showed 297 hits on that policy in about two minutes. A sample showed the culprit:

98 syscall kube-system/cilium-envoy-dxfj7 /usr/bin/cilium-envoy-starter __x64_sys_bpf

That was 98 in 15 seconds, all from cilium-envoy-starter. Tetragon never exported the events (kube-system is filtered). But the kernel still generated them, so cilium-envoy-starter joined the exclusion list.

Thirty minutes after the tuning, Tetragon had exported zero events.

Alerting on Tetragon from VictoriaLogs

Events that nobody reads are not detection. Grafana already has VictoriaLogs as a datasource. The plugin (victoriametrics-logs-datasource 0.32.0) supports alerting with queryType: stats, which returns one numeric series per group. I tested the query on 30 hours of data first:

_time:5m k8s.container.name:export-stdout process_kprobe
  | extract "\"policy_name\":\"<policy>\""
  | extract "\"namespace\":\"<namespace>\""
  | stats by (policy, namespace) count() events

# {"policy":"unexpected-egress","namespace":"databases"} 321
# {"policy":"persistence-writes","namespace":"pihole"} 5

The rule Tetragon policy hit fires on any count above zero in five minutes, with the policy and namespace as labels. It started to fire within a minute after it went live. It fired on the events from before the tuning landed:

logger=ngalert.sender.router rule_uid=tetragon-policy-hit msg="Sending alerts to local notifier" count=2

The email did not go out, because SMTP is not configured yet. That is a known gap. Until I fix it, the alert is visible only in Grafana.

Enforcement waits. The plan is about two weeks of monitor mode. Then Sigkill applies only to clear-cut cases such as kernel module loads, ptrace and writes to /etc/shadow. Shells in openbao and databases stay allowed, because admin work there goes through kubectl exec ... sh -c.

Flux in Headlamp

Flux has no UI of its own here. Headlamp has a Flux plugin. The chart has a pluginsManager that installs plugins from Artifact Hub in a sidecar:

config:
  watchPlugins: true
pluginsManager:
  enabled: true
  baseImage: node:24-alpine
  version: 0.1.1 # @headlamp-k8s/pluginctl
  configContent: |
    plugins:
      - name: flux
        source: https://artifacthub.io/packages/headlamp/headlamp-plugins/headlamp_flux
        version: 0.7.0

1 of 1 (flux): success: Plugin Installed
info: Bulk installation completed: {"total":1,"failed":0,"skipped":0,"successful":1}
Watching /config/plugin.yml for changes...

The sidecar keeps running to watch its config, and it uses 95Mi. Headlamp itself uses 18Mi. Sync, suspend and resume in the UI run with my login token. So they are real Flux operations, and protect-flux-managed reports a suspend like any other drift. One trade-off: every pod start downloads the plugin, so the pod needs internet access. Headlamp still works without the plugin.

Postgres for everything (the first four)

The idea behind postgresforeverything.com is simple. One well-run Postgres covers most of the jobs that people add Redis, RabbitMQ or etcd for. I started with four patterns that need nothing beyond core Postgres and hstore:

The database

Everything is declarative and goes through Flux:

• Role: homelab, a CNPG managed role. I generated its password straight into OpenBao at secret/databases/homelab (piped over stdin, never printed). An ExternalSecret with the cnpg.io/reload label syncs it.
• Database: a CNPG Database object homelab, with hstore turned on declaratively:

apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: homelab
spec:
  cluster:
    name: pg
  name: homelab
  owner: homelab
  extensions:
    - name: hstore
      ensure: present

roles: {"reconciled":["homelab"], ...}
database: {"applied":true,"extensions":[{"applied":true,"name":"hstore"}]}

The README says plainly that this is best effort, no SLA. There is one cluster and no backups yet, and the whole lab powers off overnight. That is fine for a lab. Anything that I cannot lose or rebuild does not go here until backups exist.

Smoke test

I ran it as the homelab role, over TCP, on the primary (pg-2 after an earlier failover):

queue ok: dequeued {"n": 1}
kv ok: a=2
hstore ok: b=2
cache ok: {"c": 31}
cache unlogged: true
Asynchronous notification "smoke" with payload "hello" received from server process with PID 4297.

There are two caveats, and both are in the README. NOTIFY is fire and forget. A disconnected listener misses messages. So pair it with the queue table when delivery matters. Also, Postgres does not replicate UNLOGGED tables, and it empties them after a crash or failover. That is fine for a cache and wrong for anything else.

KEDA stays installed. Its postgresql scaler can scale queue workers on a count(*) over the jobs table, down to zero. That will be its first job when I move my own services onto the cluster.

Result

Update: working through the fix list

Later the same day I went through the workloads left on the fix list:

Two things went wrong on the way.

A duplicate YAML key. I appended a second top-level sidecar: block to the Grafana values for the sidecar limits. YAML keeps only the last one. So the dashboards and alerts settings vanished, and Helm deleted the dashboards ConfigMap. The new pod hung in Init:

MountVolume.SetUp failed for volume "sc-dashboard-provider" : configmap "grafana-config-dashboards" not found

The old pod kept serving. I merged the keys, and that fixed it. Now I check every values file for duplicate keys before each commit.

A field that nothing removes. After the fix, the Helm values said initChownData.enabled: false. But the live Deployment still had init-chown-data. The managed fields explained it:

helm             Apply 2026-10-14T00:58:56Z  owns init-chown-data
helm-controller  Apply 2026-10-14T09:31:06Z  doesn't

The Helm CLI added the container before Flux took over (Part 16). Server-side apply removes only the fields that the applying manager owns. So helm-controller left it alone. First a dry-run showed that only the init container goes away. Then I applied an empty Deployment as the old manager, and that released its fields:

printf 'apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: grafana\n  namespace: monitoring\n' \
  | kubectl apply --server-side --field-manager=helm -f -

After the next Kyverno background scan, failures were down to 118 from 566 that morning. Only four were live. One was the Pi-hole Deployment template. The chart gives no way to set seccomp on it. The others were from the ARC listener pod. A Flux postRenderers patch adds seccompProfile: RuntimeDefault to Pi-hole. The listener now runs as uid 65532 with everything dropped.

The other 114 were old ReplicaSets: rollout history, scaled to zero. They still carried the results of their old pod templates. Kyverno now skips ReplicaSets (resourceFiltersInclude: ["[ReplicaSet,*,*]"]), because Kyverno checks Deployments and Pods anyway. The filter does not delete existing reports, so I removed 126 stale ones. Two of them stayed gone only after I restarted the reports and background controllers. The background scan still picks up a few ReplicaSets despite the filter.

Final count: 2 failures. Both are a missing memory limit on old ReplicaSets from before Part 15, and nothing runs from them. They will age out of the rollout history.

I added the SMTP password the same afternoon and tested the alert path end to end. The test email of the contact point arrived. Then a throwaway pod ran cat /etc/shadow, and Grafana sent the Tetragon policy hit alert with no delivery errors. One gotcha: Grafana 13 removed the old receiver test endpoint. The new one is /apis/notifications.alerting.grafana.app/v1beta1/namespaces/default/receivers/<uid>/test and needs an integration body.

Update 2026-10-19: the guardrails now block

The tuning here made the reports readable enough to act on. Part 34 used them. Three Tetragon detections that never fired now kill. The two Kyverno policies with clean reports now deny. The :latest image of the CI runner was the last hygiene offender. See Part 34.

Lessons

1. An audit log that nobody reads is not a control. The first job was to remove expected noise, so that the remaining failures mean something.
2. Exempt the narrowest thing that works. Whole namespaces only for system components. One DaemonSet for the OTel agent. Everything else stays on the fix list.
3. The status of a policy can lag its permissions. After I fixed the RBAC, the policy became ready only after a controller restart.
4. Count before you alert. Two sources made up most of the Tetragon events (IPv6 loopback and the restore_command of Postgres itself), and both were benign. An alert first means an alert on noise.
5. Check what a new hook catches in the kernel, not only in the export. The bpf() hook looked quiet in VictoriaLogs. But it fired six times a second on a filtered namespace.
6. Say the reliability out loud. A shared database with no backups is useful, but only if the README says so before anyone depends on it.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep (this post)
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• Kyverno: Policy Exceptions
• Kyverno: ValidatingPolicy
• Tetragon: Tracing Policy
• Tetragon: Filtering events
• VictoriaLogs: Grafana datasource
• Headlamp: Flux plugin
• CloudNativePG: Declarative database management
• CloudNativePG: Declarative role management
• PostgreSQL: SELECT, the locking clause (SKIP LOCKED)
• PostgreSQL: NOTIFY
• Postgres for Everything