Homelab Rebuild Part 28: Observability, Take Three: Making It Worth Looking At

Intro

Since Part 14, the cluster had OpenTelemetry collectors, VictoriaMetrics, VictoriaLogs and Grafana. It worked, in the sense that data went in. But it was not useful. The logs dashboard was a wall of noise. There was nothing for Postgres, OpenBao or Flux, and nothing told me when the gateway was down. It went down three times on 2026-10-17 alone, about a minute each. I knew only because I was watching.

This post covers one evening of work to fix that, and the two short outages that I caused during the work.

TL;DR: 90% of all log lines (13.1M of 14.5M a day) came from a cilium-monitor debug sidecar that the very first cilium install added and Flux never removed. An OTel filter processor took the log rate from about 10,000 lines a minute to about 150. I added new dashboards for CloudNativePG, OpenBao, Flux and the gateway/edge. I also added synthetic HTTPS probes (scripted requests on a timer) and TLS expiry, with alerts. There were two self-inflicted outages. A corrupt cached image took tailnet DNS down for 3 minutes. A half-applied TLS switch (server-side apply merged two probe handlers) left Grafana with no data for about 9 minutes.

The Figure

The figure is a strip-chart recorder. Log lines enter at the intake on the left. The mesh screen is the OTel filter, and what it catches drops into the chaff tray. Each pen is one dashboard (CloudNativePG, OpenBao, Flux, gateway and edge). The seven round gauges are the HTTPS probes, and the day drum shows how long the TLS certificates have left. The slider on the front replays the HTTPS switch from Issue 5. The pens move to HTTPS before the paper feed does, and the strip runs blank until both match. Press F for the filter, H for HTTPS, C for the certificates and P for the probes. You can also click the left, middle or right of the figure. R resets. Drawn with the anatomy skill (GitHub: wheresryan22/anatomy).

The Situation

• k0s 1.36.4, 3 controllers and 5 workers on Proxmox (pve-01, pve-02, pve-03). Flux owns everything.
• Cilium 1.20.2 with Gateway API (lab-gateway on 192.168.101.152), WireGuard between nodes.
• OTel collector 0.162.0 (otelcol-k8s image, chart 0.176.0): a DaemonSet agent for pod logs and kubelet metrics, and one cluster collector for Kubernetes objects and events.
• VictoriaMetrics (30 days), VictoriaLogs (14 days), Grafana 13.5.

flowchart LR
  P["pod logs, kubelet"] --> A["OTel agent<br/>(filter/noise)"]
  S["Postgres, Envoy,<br/>cloudflared, cert-manager,<br/>Tailscale"] -->|scrape| C["OTel cluster collector"]
  H["HTTPS probes<br/>every 30 s"] --> C
  A -->|"OTLP over HTTPS"| VM["VictoriaMetrics"]
  A -->|"OTLP over HTTPS"| VL["VictoriaLogs"]
  C --> VM
  C --> VL
  VM --> G["Grafana<br/>dashboards, alerts"]
  VL --> G

Issue 1: 90% of the Logs Were One Sidecar

Symptom

The logs dashboard was unreadable. I counted lines per namespace over 24 hours in VictoriaLogs:

Inside kube-system, per container:

Cause

cilium-monitor is a debug sidecar that runs cilium-dbg monitor and prints every network flow:

-> endpoint 229 flow 0x199bb245 , identity host->36320 state established ifindex lxccb82d9cb725f orig-ip 10.0.1.199

My Helm values never turn it on, and the chart default is monitor.enabled: false. The live DaemonSet still had it. Its managedFields owner was cilium, last updated 2026-10-12T11:10:03Z: the very first cilium install, before Flux took over. Flux upgrades with server-side apply. Server-side apply never removes a field that the chart stops rendering, because the field belongs to a different field manager. It is the same trap as the stale enable-ingress-controller key in Part 26. This time it was a whole container.

konnectivity-agent was the other 7%. k0s ships it with a log line for each tunneled packet ("received DATA", "write to remote", "Determined server count").

Fix

The fix has two parts. First, drop the noise at the agent. Then it never reaches VictoriaLogs, whatever runs in the cluster:

processors:
  filter/noise:
    error_mode: ignore
    logs:
      log_record:
        - 'resource.attributes["k8s.container.name"] == "cilium-monitor"'
        - 'resource.attributes["k8s.container.name"] == "konnectivity-agent" and IsMatch(body, "(received DATA|write to remote|Determined server count)")'

Lines per minute, all namespaces:

Second, remove the sidecar itself, by hand, once:

kubectl -n kube-system patch ds cilium --type=json \
  -p '[{"op":"remove","path":"/spec/template/spec/containers/1"}]'

A patch to the DaemonSet restarts every cilium agent. Every agent restart takes lab-gateway down until Envoy gets its listeners back. This time the outage was 51 seconds (22:12:02 to 22:12:53 WIB).

Issue 2: Dashboards That Answer Questions

Postgres

CloudNativePG exposes metrics on a port named metrics (9187), but nothing scraped it. The cluster collector now scrapes it with Kubernetes service discovery. It keeps the instance role as a label:

receivers:
  prometheus/apps:
    config:
      scrape_configs:
        - job_name: cnpg
          kubernetes_sd_configs:
            - role: pod
              namespaces: {names: [databases]}
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_container_port_name]
              regex: metrics
              action: keep
            - source_labels: [__meta_kubernetes_pod_label_cnpg_io_instanceRole]
              target_label: role

The dashboard shows instances up, streaming replicas and replication lag. It shows which instance is primary (a flip means a failover). It also shows transactions per second, connections, database size, cache hit ratio and WAL. I found two surprises on the way:

• Counter names change. VictoriaMetrics runs with opentelemetry.usePrometheusNaming, so counters get _total (cnpg_pg_stat_database_xact_commit_total) and units become suffixes (httpcheck_duration_milliseconds). My first dashboard used the exporter’s names and showed nothing for TPS. Check names in the database, not in the exporter.
• Postgres logs are JSON. A regex for error matches the "error_severity" key on every line. So I matched the value instead: "error_severity":"(WARNING|ERROR|FATAL|PANIC)". That found 4,153 warning and error lines in 48 hours that I did not see before. Also: inside a LogsQL quoted string, \b is a backspace, not a regex word boundary.

OpenBao and Flux

OpenBao has no metrics scrape yet. So its dashboard uses logs and kubelet metrics: pods ready, restarts, logins by method, seal and Raft lines, and permission denied. I count logins from lease revocations: kubernetes is External Secrets, and oidc is me. Flux’s dashboard reads Kubernetes events from VictoriaLogs: Warning events per kind, a failures table and the applied revisions. Events have no _msg field there, so the panels select object.reason, object.note and object.regarding.* directly.

Issue 3: Watching the Gateway and the Edge

Probes

The OTel httpcheck receiver probes every 30 seconds: the apps behind lab-gateway (with the lab CA verified), the OpenBao API and the three Proxmox UIs. One catch: pods cannot resolve *.franzramadhan.lab, because CoreDNS forwards to the node’s resolver, not Pi-hole. I did not edit the k0s-managed CoreDNS. Instead, the collector pod gets hostAliases that point the probed names at .152.

For the “down” alert, I read the receiver source first. It records httpcheck.duration before it checks for a request error, so a dead endpoint still produces a duration sample. So “probed, but no 2xx for 2 minutes” is a reliable down signal:

count by (http_url)(max_over_time(httpcheck_duration_milliseconds[2m]))
  unless on(http_url)
(max by (http_url)(max_over_time(httpcheck_status_ratio{http_status_class="2xx"}[2m])) == 1)

A test with the class flipped to 5xx fired for all 7 URLs.

Envoy, cloudflared and Tailscale

The same Prometheus receiver scrapes three more targets:

• cilium-envoy (envoy-metrics, 9964), for requests, 5xx and p95 latency per gateway backend.
• cloudflared (2000), for its 4 HA connections per pod.
• The Tailscale proxies, after I enabled metrics.enable in the default ProxyClass.

TLS expiry

I use two sources, because they answer different questions. The first is cert-manager’s certmanager_certificate_expiration_timestamp_seconds (every Certificate it manages). The second is the receiver’s httpcheck.tls.cert_remaining. That is the certificate each endpoint actually serves, including the Proxmox UIs that cert-manager does not manage. Days left at 22:25 WIB:

cert-manager renews at 30 days, so the alert fires below 14. At that point, renewal failed for two weeks.

Issue 4: A Metrics Toggle Took Tailnet DNS Down

Symptom

When you enable ProxyClass metrics, the Tailscale proxies restart. The one that serves tailnet DNS came back as Init:Error:

# kubectl -n tailscale logs ts-pihole-dns-tailscale-...-0 -c sysctler
exec /bin/sh: exec format error

Cause

The image (tailscale/tailscale:v1.102.4) was “already present on machine” on k0s-pve-01-worker-1, and the same image ran fine on both pve-02 workers. If a binary suddenly has the wrong format on one node, the cause is a corrupt cached image, not a bad release.

Fix

k0s crictl rmi docker.io/tailscale/tailscale:v1.102.4   # on the worker
kubectl -n tailscale delete pod ts-pihole-dns-tailscale-...-0

The pod came back on another worker, and tailnet DNS answered 10 of 10 queries. DNS was down for about 3 minutes (15:21 to 15:24 UTC). During that time, tailnet devices fell back to 1.1.1.3.

Update 2026-10-18: this fix was wrong twice. k0s 1.36 has no crictl subcommand (the command is k0s ctr -n k8s.io images rm ...). Also, deletion of the image does not remove its unpacked layer. The pod recovered only because it moved to another worker. The layer on k0s-pve-01-worker-1 held 0-byte binaries since the Oct 13 crash. The real fix, ctr snapshots rm on the layer, is in Part 29.

Issue 5: The TLS Switch That Applied Halfway

Symptom

VictoriaMetrics and VictoriaLogs served plain HTTP inside the cluster. WireGuard already encrypts node to node, but I wanted HTTPS end to end. One PR added a lab-CA Certificate and turned on --tls for both. It also switched the two collectors and Grafana’s data sources to https with the CA. After the merge, every Grafana panel said “No data”.

Cause

Grafana and the collectors upgraded. The two databases did not:

StatefulSet.apps "vmsingle" is invalid:
spec.template.spec.containers[0].readinessProbe.tcpSocket:
Forbidden: may not specify more than 1 handler type

With TLS on, the VictoriaMetrics chart switches the readiness probe from httpGet to tcpSocket. Server-side apply merges the new manifest into the live object field by field. So it kept the live httpGet and added tcpSocket, and the API rejects a probe with two handlers. As a result, the clients spoke HTTPS to servers that were still on HTTP.

sequenceDiagram
  participant Flux
  participant G as Grafana and collectors
  participant V as VictoriaMetrics/Logs
  Flux->>G: upgrade: use https
  G-->>Flux: ok
  Flux->>V: upgrade: --tls, tcpSocket probe
  V-->>Flux: rejected (httpGet + tcpSocket)
  G->>V: HTTPS to an HTTP server
  V-->>G: fails: "No data"

Fix

One upgrade with a client-side three-way merge fixed it. That merge deletes httpGet, because the previous release manifest had it:

spec:
  upgrade:
    serverSideApply: disabled   # one-off, reverted in the next PR

Both StatefulSets rolled, and Grafana’s data sources reported OK over HTTPS. VictoriaLogs took 9,472 lines in the first minute while the collectors flushed their queues. The gap was about 9 minutes (15:32 to 15:41 UTC). Data older than the retry window of the exporters is gone.

Update 2026-10-19: control plane and kubelet logs

One gap was left after this post. Pod logs went to VictoriaLogs, but the control plane logs did not. kube-apiserver, etcd, the scheduler and the controller manager run inside k0scontroller.service on the controllers, and the kubelet inside k0sworker.service on the workers. Their logs stayed in each VM’s journal.

Three things ruled out the obvious paths:

• The k0s controllers are not Kubernetes nodes, so the OTel agent DaemonSet never runs there.
• The agent image is distroless, with no journalctl for the collector’s journald receiver.
• VictoriaLogs sits behind a headless ClusterIP Service, out of reach from the VMs.

But VictoriaLogs can ingest the journal natively. So the VMs push it themselves with systemd-journal-upload. lab-gateway exposes exactly one path for that, /insert/journald, with TLS to the backend verified against the lab CA. The query API is not routed:

VM journal -> systemd-journal-upload -> https://vlogs-ingest.franzramadhan.lab:443/insert/journald
           -> lab-gateway (only /insert/journald) -> vlogs:9428

Two details from the install:

• Without a port, systemd-journal-upload appends its default :19532 after the path, and every upload got a 404: Upload to https://vlogs-ingest.franzramadhan.lab/insert/journald:19532/upload failed with code 404. The URL needs :443 spelled out.
• To skip days of old history, I write the current cursor to the state file before the first start: LAST_CURSOR=$(journalctl -n0 --show-cursor ...).

The first night

The lab shuts down at 23:30 and starts at 06:00, so the first night tested the restart path on all eight VMs:

The duplicates come from systemd-journal-upload, which delivers at least once. On restart, it resends the last batch that it did not record as sent. That is harmless for searches, but know it before you count lines.

The first real question was whether the control plane complained overnight. One LogsQL query across all three controllers:

_time:24h _SYSTEMD_UNIT:k0scontroller.service ("level=error" OR "level=warning")
  | extract "msg=\"<m>\"" | stats by (m) count() n

It returned thirty-nine lines in 24 hours, all from the shutdown and the boot. They were terminating processes, the controller lease that changed hands, and etcd gRPC retries while its peers came up. In steady state, there were none.

Lessons

1. Look at log volume per container before you build log dashboards. One forgotten sidecar was 90% of everything.
2. Fields from an old installer survive Flux forever. Server-side apply does not delete what another field manager owns: a ConfigMap key in Part 26, a whole container here.
3. A chart change that swaps a field’s type breaks server-side apply. Render it, expect the merge to fail, and plan the one-off client-side upgrade in the same change. Upgrade servers before clients, and check every HelmRelease after a merge, not only the first one.
4. Check metric names where they are stored. The translation to Prometheus naming changed them, and the dashboard silently showed nothing.
5. A harmless toggle is still a restart. When I enabled metrics, a DNS proxy restarted, and the restart found a corrupt image cache.

Series

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At (this post)
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• GitHub: wheresryan22/anatomy
• OpenTelemetry: Filter processor
• OpenTelemetry: HTTP Check receiver
• CloudNativePG: Monitoring
• VictoriaMetrics: OpenTelemetry ingestion
• VictoriaLogs: LogsQL
• Kubernetes: Server-Side Apply
• Flux: HelmRelease upgrade configuration
• Tailscale: Customizing the operator with ProxyClass