Homelab Rebuild Part 17: Making the Guardrails Earn Their Keep

Intro

With Flux in charge (Part 16), I went through the cluster and asked a simple question of every component: is anyone using this? Several answers were no. Kyverno had 21 policies, all in Audit, producing 566 failures nobody read. Tetragon had five detection policies writing events nobody looked at. KEDA had zero ScaledObjects. The CloudNativePG cluster had zero client connections.

This post covers making the first two useful, adding a Flux view to Headlamp, and giving Postgres its first job.

TL;DR: Most Kyverno failures were system pods that need privileges by design. Two PolicyExceptions took the failures from 566 to 246, and what’s left is a real fix list. Three new rules target mistakes I actually made, starting with refusing anything in default. Tetragon’s top two noise sources were IPv6 loopback (::1) and Postgres’ own restore_command. After tuning, it exported zero events in 30 minutes, and a Grafana alert on VictoriaLogs now watches for policy hits. Postgres got a homelab database for queues, pub/sub, key/value and cache, explicitly best effort with no SLA.

The Situation

• k0s cluster on three Proxmox hosts, Flux reconciling 20 apps from Git
• Kyverno 1.19.1 (chart 3.9.1): the chart’s 17 Pod Security Standards policies plus my own from Part 11, all ValidatingPolicy (CEL), mostly Audit
• Tetragon 1.7.1: five policies in monitor mode, events exported to stdout, collected by the OTel agent into VictoriaLogs
• CloudNativePG 1.30.1, cluster pg with two instances, no backups yet
• Headlamp 0.45.0 as the cluster UI
• Decisions going in: keep KEDA for future queue workers, skip backups until I pick an object store, and use Postgres for everything instead of adding Redis or a message broker

Issue 1: 566 Kyverno failures, almost all expected

Symptom

kubectl get policyreports -A -o json | jq -r '.items[].results[]? | select(.result=="fail") | .policy' | sort | uniq -c | sort -rn | head -5
# 43 restrict-seccomp-strict
# 40 disallow-host-namespaces
# 29 require-run-as-nonroot
# 28 disallow-host-ports
# 24 disallow-capabilities-strict

283 reports, 566 failures, and the top entries were all in kube-system: Cilium, Tetragon and the k0s components. They need host networking, host paths and capabilities to do their job. With that much expected red, a real problem in an app namespace would never stand out.

Cause

The kyverno-policies chart has policyExclude for exactly this, but in CEL mode (ValidatingPolicy) the chart ignores it. Exclusions have to be PolicyException objects, and that feature was off. A server-side dry-run of my first exception said so:

Warning: PolicyException resources would not be processed until it is enabled.

Fix

Enable exceptions, but only from the kyverno namespace, so an app namespace can’t exempt itself:

features:
  policyExceptions:
    enabled: true
    namespace: kyverno

Then two exceptions. One exempts whole system namespaces from the 17 PSS policies. The other exempts only the OTel agent DaemonSet, which reads host logs and metrics:

apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: system-namespaces
  namespace: kyverno
spec:
  policyRefs:
    - {kind: ValidatingPolicy, name: disallow-host-namespaces}
    - {kind: ValidatingPolicy, name: disallow-host-path}
    # ... the other 15 PSS policies
  matchConditions:
    - name: system-namespace
      expression: >-
        object.metadata.namespace in ["kube-system", "tailscale", "local-path-storage"]

Everything else stays reported on purpose: Grafana, vmsingle, the ARC controller, Pi-hole and others now make up a fix list, not a list of exceptions.

The Helm upgrade itself hiccupped. Release v5 was left in pending-upgrade, Flux unlocked it, and v6 succeeded. The reports then took a while to catch up:

The 66 failures still listed in exempt namespaces at that point all had timestamps from before the exceptions existed. They are old results waiting for the next background scan, not new failures.

Issue 2: New rules, and a policy that stayed not ready

I added rules aimed at mistakes I had actually made:

The deny works:

kubectl run deny-test --image=busybox:1.37 -n default --dry-run=server
# Error from server: admission webhook "vpol.validate.kyverno.svc-ignore-9d71e4df" denied the request:
# Policy deny-default-namespace failed: nothing runs in the default namespace: set metadata.namespace (or use the app's kustomization)

Symptom

deny-default-namespace showed READY false, even though it was already blocking requests:

RBACPermissionsGranted False Failed Policy is not ready for reporting, missing permissions:
get postgresql.cnpg.io/v1, Resource=clusters: ; list ...; watch ...

Cause

The policy also matches CNPG Cluster objects, and Kyverno’s reports controller had no permission to read them. Enforcement only needs the admission webhook, so blocking was unaffected; only the reporting was missing.

Fix

A read-only ClusterRole that Kyverno’s aggregation label merges into the reports controller:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:reports-controller:cnpg
  labels:
    rbac.kyverno.io/aggregate-to-reports-controller: "true"
rules:
  - apiGroups: ["postgresql.cnpg.io"]
    resources: ["clusters"]
    verbs: ["get", "list", "watch"]

The role was aggregated correctly, but the policy stayed READY false: the condition’s timestamp never moved. Kyverno only rechecks permissions when the policy changes, so I restarted the admission and reports controllers. After that, all 24 policies were ready.

A rule that flagged me first

protect-flux-managed matches every resource type, which worried me: if Kyverno didn’t pass the label filter through to its webhook, every update in the cluster would go through Kyverno. It did pass it through:

kyverno-resource-validating-webhook-cfg vpol.validate.kyverno.svc-ignore-7e7ae6df
  {"matchExpressions": [{"key": "kustomize.toolkit.fluxcd.io/name", "operator": "Exists"}]}

Its first report was me: the reconcile.fluxcd.io/requestedAt annotation I set to make Flux reconcile immediately. flux reconcile and Headlamp’s sync button do the same. Updates that leave the spec and labels untouched now pass:

- name: annotationOnly
  expression: >-
    request.operation == "UPDATE" &&
    object.?spec == oldObject.?spec &&
    object.metadata.?labels == oldObject.metadata.?labels

Suspending an app from Headlamp changes spec.suspend and is still reported, which is the point: that is drift from Git.

Issue 3: Tetragon was mostly watching Postgres

Symptom

Tetragon events were reaching VictoriaLogs, so I counted them by type, policy and binary. About 13 hours of data:

321 process_kprobe  unexpected-egress  databases  /controller/manager
295 process_exec    databases          /bin/sh
 24 process_exec    default            /bin/sh
  5 process_kprobe  persistence-writes pihole     /usr/bin/crontab

The 24 shells in default were the duplicate Postgres cluster from Part 15. The rest needed a closer look.

Cause

The egress policy reports outbound connections from databases and openbao to anything outside the cluster, LAN and loopback ranges. 315 of the 321 events went to the CNPG instance manager’s own status port, over IPv6:

{"d":"::1","port":"8010","n":"315"}

The allow list had 127.0.0.0/8 but not ::1.

The 295 shells came from standby pg-1. Their parent was postgres, and the command was CNPG’s restore_command:

sh -c -- "/controller/manager wal-restore --log-destination /controller/log/postgres.json 00000005.history pg_wal/RECOVERYHISTORY"

A standby runs it for every WAL file and timeline it looks for. All 295 ran between 02:41 and 12:11 WIB, a window with two outages and their recoveries, and none in the two hours before I looked.

The five crontab writes were Pi-hole managing its own schedule.

Fix

Add ::1/128 to both egress allow lists, and drop the two known-benign repeats at export, matched as narrowly as Tetragon’s filters allow:

exportDenyList: |-
  {"health_check":true}
  {"namespace":["","kube-system","tailscale","local-path-storage"]}
  {"namespace":["databases"],"binary_regex":["^/bin/sh$"],"arguments_regex":["/controller/manager wal-restore "]}
  {"namespace":["pihole"],"binary_regex":["^/usr/bin/crontab$"],"policy_names":["persistence-writes"]}

My first version had a # comment line inside that block. It’s JSON lines, not YAML, so the comment would have broken the filter parsing. I caught it before committing and moved the comment above the key.

I also added a bpf() hook to the privilege escalation policy, since loading eBPF programs from a pod is a classic escape primitive. Cilium and Tetragon load them legitimately and are excluded. Right after the rollout, tetra tracingpolicy list showed 297 hits on that policy in about two minutes. A sample showed the culprit:

98 syscall kube-system/cilium-envoy-dxfj7 /usr/bin/cilium-envoy-starter __x64_sys_bpf

That was 98 in 15 seconds, all from cilium-envoy-starter. The events were never exported (kube-system is filtered), but they were still generated in the kernel, so it joined the exclusion list.

Thirty minutes after the tuning, Tetragon had exported zero events.

Alerting on Tetragon from VictoriaLogs

Events nobody reads aren’t detection. Grafana already has VictoriaLogs as a datasource, and the plugin (victoriametrics-logs-datasource 0.32.0) supports alerting with queryType: stats, which returns one numeric series per group. I tested the query on 30 hours of data first:

_time:5m k8s.container.name:export-stdout process_kprobe
  | extract "\"policy_name\":\"<policy>\""
  | extract "\"namespace\":\"<namespace>\""
  | stats by (policy, namespace) count() events

# {"policy":"unexpected-egress","namespace":"databases"} 321
# {"policy":"persistence-writes","namespace":"pihole"} 5

The rule Tetragon policy hit fires on any count above zero in five minutes, labelled with the policy and namespace. It started firing within a minute of going live, on the events from before the tuning landed:

logger=ngalert.sender.router rule_uid=tetragon-policy-hit msg="Sending alerts to local notifier" count=2

The email didn’t go out, because SMTP isn’t configured yet. That’s a known gap; until it’s fixed, the alert is visible only in Grafana.

Enforcement waits. The plan is about two weeks of monitor mode, then Sigkill only for clear-cut cases such as kernel module loads, ptrace and writes to /etc/shadow. Shells in openbao and databases stay allowed, because admin work there goes through kubectl exec ... sh -c.

Flux in Headlamp

Flux has no UI of its own here. Headlamp has a Flux plugin, and the chart has a pluginsManager that installs plugins from Artifact Hub in a sidecar:

config:
  watchPlugins: true
pluginsManager:
  enabled: true
  baseImage: node:24-alpine
  version: 0.1.1 # @headlamp-k8s/pluginctl
  configContent: |
    plugins:
      - name: flux
        source: https://artifacthub.io/packages/headlamp/headlamp-plugins/headlamp_flux
        version: 0.7.0

1 of 1 (flux): success: Plugin Installed
info: Bulk installation completed: {"total":1,"failed":0,"skipped":0,"successful":1}
Watching /config/plugin.yml for changes...

The sidecar stays running to watch its config, using 95Mi. Headlamp itself uses 18Mi. Sync, suspend and resume in the UI run with my login token, so they are real Flux operations, and protect-flux-managed reports a suspend like any other drift. One trade-off: every pod start downloads the plugin, so it needs internet access. Headlamp still works without the plugin.

Postgres for everything (the first four)

The idea behind postgresforeverything.com is that one well-run Postgres covers most of what people add Redis, RabbitMQ or etcd for. I started with four patterns that need nothing beyond core Postgres and hstore:

The database

Everything is declarative and goes through Flux:

• Role: homelab, a CNPG managed role. Its password was generated straight into OpenBao at secret/databases/homelab (piped over stdin, never printed), then synced by an ExternalSecret with the cnpg.io/reload label.
• Database: a CNPG Database object homelab, with hstore turned on declaratively:

apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: homelab
spec:
  cluster:
    name: pg
  name: homelab
  owner: homelab
  extensions:
    - name: hstore
      ensure: present

roles: {"reconciled":["homelab"], ...}
database: {"applied":true,"extensions":[{"applied":true,"name":"hstore"}]}

The README says plainly that this is best effort, no SLA: one cluster, no backups yet, and the whole lab powers off overnight. That’s fine for a lab. Anything I can’t lose or rebuild doesn’t go here until backups exist.

Smoke test

As the homelab role, over TCP, on the primary (pg-2 after an earlier failover):

queue ok: dequeued {"n": 1}
kv ok: a=2
hstore ok: b=2
cache ok: {"c": 31}
cache unlogged: true
Asynchronous notification "smoke" with payload "hello" received from server process with PID 4297.

There are two caveats, and both are in the README. NOTIFY is fire and forget: a disconnected listener misses messages, so pair it with the queue table when delivery matters. And UNLOGGED tables aren’t replicated and are emptied after a crash or failover. That’s fine for a cache and wrong for anything else.

KEDA stays installed. Its postgresql scaler can scale queue workers on a count(*) over the jobs table, down to zero. That will be its first job once I move my own services onto the cluster.

Result

Update: working through the fix list

Later the same day I went through the workloads left on the fix list:

Two things went wrong on the way.

A duplicate YAML key. I appended a second top-level sidecar: block to Grafana’s values for the sidecar limits. YAML keeps only the last one, so the dashboards and alerts settings vanished, the dashboards ConfigMap was deleted, and the new pod hung in Init:

MountVolume.SetUp failed for volume "sc-dashboard-provider" : configmap "grafana-config-dashboards" not found

The old pod kept serving. Merging the keys fixed it, and I now check every values file for duplicate keys before committing.

A field nobody would remove. With the fix applied, Helm’s values said initChownData.enabled: false, but the live Deployment still had init-chown-data. The managed fields explained it:

helm             Apply 2026-10-14T00:58:56Z  owns init-chown-data
helm-controller  Apply 2026-10-14T09:31:06Z  doesn't

The container was added by the Helm CLI before Flux took over (Part 16). Server-side apply only removes fields that the applying manager owns, so helm-controller left it alone. Applying an empty Deployment as the old manager released its fields, after a dry-run showed only the init container would go:

printf 'apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: grafana\n  namespace: monitoring\n' \
  | kubectl apply --server-side --field-manager=helm -f -

After Kyverno’s next background scan, failures were down to 118 from 566 that morning. Only four were live: Pi-hole’s Deployment template, which the chart gives no way to set seccomp on, and the ARC listener pod. A Flux postRenderers patch adds seccompProfile: RuntimeDefault to Pi-hole, and the listener now runs as uid 65532 with everything dropped.

The other 114 were old ReplicaSets: rollout history, scaled to zero, still carrying the results of their old pod templates. Kyverno now skips ReplicaSets (resourceFiltersInclude: ["[ReplicaSet,*,*]"]), since Deployments and Pods are checked anyway. The filter doesn’t delete existing reports, so I removed 126 stale ones, and two only stayed gone after restarting the reports and background controllers. The background scan still picks up a few ReplicaSets despite the filter, so the filter works only partly there.

Final count: 2 failures, both a missing memory limit on old ReplicaSets from before Part 15 that nothing runs from. They’ll age out of the rollout history.

The SMTP password went in the same afternoon, and I tested the alert path end to end: the contact point’s test email arrived, then a throwaway pod ran cat /etc/shadow and Grafana sent the Tetragon policy hit alert with no delivery errors. One gotcha: Grafana 13 removed the old receiver test endpoint; the new one is /apis/notifications.alerting.grafana.app/v1beta1/namespaces/default/receivers/<uid>/test and needs an integration body.

Lessons

1. An audit log nobody reads is not a control. The first job was to remove expected noise so the remaining failures mean something.
2. Exempt the narrowest thing that works. Whole namespaces only for system components; one DaemonSet for the OTel agent; everything else stays on the fix list.
3. A policy’s status can lag its permissions. After the RBAC was fixed, the policy only became ready after a controller restart.
4. Count before you alert. Two sources made up most of Tetragon’s events (IPv6 loopback and Postgres’ own restore_command), and both were benign. Alerting first would have meant alerting on noise.
5. Check what a new hook catches in kernel, not just in the export. The bpf() hook looked quiet in VictoriaLogs and was firing six times a second on a filtered namespace.
6. Say the reliability out loud. A shared database with no backups is useful, as long as the README says so before anyone depends on it.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep (this post)
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google

References

• Kyverno: Policy Exceptions
• Kyverno: ValidatingPolicy
• Tetragon: Tracing Policy
• Tetragon: Filtering events
• VictoriaLogs: Grafana datasource
• Headlamp: Flux plugin
• CloudNativePG: Declarative database management
• CloudNativePG: Declarative role management
• PostgreSQL: SELECT, the locking clause (SKIP LOCKED)
• PostgreSQL: NOTIFY
• Postgres for Everything