Homelab Rebuild Part 14: Observability, Take Two with OpenTelemetry

Intro

Coroot (Part 12) lasted one day. While pve-01 kept losing power (Part 13), I suspected load, and the heaviest thing in the cluster was Coroot: an eBPF agent of about 450 MiB on every node plus ClickHouse. Load turned out not to be the cause, but the question stuck: did I need a service map, or did I need metrics, logs and alerts?

The second. So I replaced it with plain OpenTelemetry: one standard for collecting, two small VictoriaMetrics stores, and Grafana on top.

TL;DR: OpenTelemetry Collector (a DaemonSet agent plus a cluster collector) sends everything over OTLP/HTTP to VictoriaMetrics and VictoriaLogs, with no Prometheus scraping anywhere. Grafana 13 reads both and loads dashboards and alert rules from ConfigMaps. The whole stack uses about 1.9 GiB, against about 3 GiB for Coroot. Watch out for host metrics without a node label, and for datasource uid changes in Grafana provisioning.

The Situation

• k0s cluster, 5 workers across pve-01, pve-02 and pve-03; the pve-03 worker is tainted dedicated=quorum:NoSchedule
• Two workers have a local-path data disk, labeled lab/data-disk=true
• Pod Security baseline by default and Kyverno policies from Part 11, including one that limits each ExternalSecret to secret/<namespace>/...
• The Proxmox hosts send metrics and logs to Grafana Cloud (free tier) through Grafana Alloy, separately from the cluster

Choosing the Stack

Both VictoriaMetrics stores accept OTLP directly, so the collector doesn’t need a Prometheus exporter or remote write. Traces have no backend yet; I’ll add VictoriaTraces once an app sends them.

The grafana/grafana chart is deprecated; the chart moved to the grafana-community repo.

Setup

Two collectors

The collector chart has presets that write most of the config. The agent runs on every node and handles what is local to it:

mode: daemonset
image:
  repository: otel/opentelemetry-collector-k8s
presets:
  logsCollection:          # /var/log/pods
    enabled: true
  kubeletMetrics:          # pod, container and node metrics
    enabled: true
  hostMetrics:             # CPU, memory, disk, filesystem, network of the VM
    enabled: true
  kubernetesAttributes:    # namespace, pod and workload names on everything
    enabled: true
tolerations:
  - key: dedicated
    operator: Equal
    value: quorum
    effect: NoSchedule

The cluster collector runs once and watches the API server: clusterMetrics (node conditions, pod phases, deployment replicas) and kubernetesEvents (events into the logs store).

Both export over OTLP/HTTP only:

exporters:
  otlphttp/metrics:
    metrics_endpoint: http://vmsingle.monitoring.svc:8428/opentelemetry/v1/metrics
  otlphttp/logs:
    logs_endpoint: http://vlogs.monitoring.svc:9428/insert/opentelemetry/v1/logs

I removed what I don’t use: the Jaeger and Zipkin receivers and ports, the prometheus receiver, and the traces pipeline.

Prometheus names in VictoriaMetrics

OTel metric names look like k8s.pod.cpu.time. With -opentelemetry.usePrometheusNaming, VictoriaMetrics stores them the way PromQL users expect, k8s_pod_cpu_time_seconds_total, with units and _total added. One trap: k8s_container_restarts arrives as a gauge, not a counter, so increase() gives nonsense. Use max_over_time(...) - min_over_time(...) instead.

Grafana from ConfigMaps

The chart’s sidecars load anything labeled grafana_dashboard or grafana_alert, so dashboards and alert rules live in Git as files:

kubectl -n monitoring create configmap grafana-dashboard-nodes --from-file=dashboards/nodes.json \
  --dry-run=client -o yaml | kubectl label --local -f - grafana_dashboard=1 -o yaml \
  | kubectl apply --server-side -f -

The admin password and SMTP login come from OpenBao through External Secrets (secret/monitoring/grafana and secret/monitoring/smtp).

Dashboards and alerts

Five dashboards, all based on what this cluster actually has:

And nine alert rules, sent by email:

“Node silent” exists because “not ready” depends on the API server still being up. A node whose agent stops sending is visible even when the cluster is in trouble.

Email goes through Gmail SMTP, which needs a Gmail App Password, not the account password.

On the first run, “Container near memory limit” already had a match: the Kyverno cleanup controller at 91% of its 128 MiB limit.

Issue 1: Host metrics from all nodes in one line

Symptom

The Nodes dashboard showed one CPU line, jumping between values, instead of one per node.

Cause

The hostMetrics preset scrapes the VM itself, and its resource has no Kubernetes attributes. Without a node label, all five agents wrote the same series, and each one overwrote the others.

Fix

Stamp the node name on everything the agent sends. The chart already sets K8S_NODE_NAME from the downward API:

processors:
  resource/node:
    attributes:
      - key: k8s.node.name
        value: ${env:K8S_NODE_NAME}
        action: upsert
service:
  pipelines:
    metrics:
      processors: [memory_limiter, resource/node, batch]

Issue 2: A metric the receiver didn’t support

Symptom

After I enabled k8s.container.status.last_terminated_reason in the cluster collector to alert on OOM kills, its pod went into CrashLoopBackOff on a config error.

Cause

The k8s_cluster receiver in collector 0.162.0 doesn’t support that metric. The collector validates its config at start and refuses to run with an unknown key.

Fix

Remove it. The “Container near memory limit” alert catches OOM risk before the kill instead of after.

Issue 3: Grafana crashed on a datasource uid

Symptom

To reference datasources from dashboard files, I added fixed uids (vm, vlogs) to the provisioned datasources. Grafana crashed on start with data source not found.

Cause

The datasources already existed in Grafana’s database with generated uids. Provisioning matched them by name, then failed to update them to the new uid.

Dead end

My first idea was to delete the rows in the SQLite database by hand. The Grafana image has no shell, and while trying I deleted the running pod, so Grafana was down for a few minutes for nothing.

Fix

Provisioning can delete datasources before it creates them:

datasources:
  datasources.yaml:
    apiVersion: 1
    deleteDatasources:
      - name: VictoriaMetrics
        orgId: 1
      - name: VictoriaLogs
        orgId: 1
    datasources:
      - name: VictoriaMetrics
        uid: vm
        ...

It runs on every start and is harmless once the uids match.

Issue 4: Small ones

• Dashboards landed in General. The sidecar puts files in the default folder unless the provider names one: sidecar.dashboards.provider.folder: Homelab.
• An ExternalSecret was rejected. I applied it without -n monitoring, so it went to default. The Kyverno policy from Part 11 only allows a namespace to read its own OpenBao path, and correctly refused.

The Proxmox Hosts

The hosts aren’t Kubernetes nodes, so the OTel stack doesn’t see them. They run Grafana Alloy and send to Grafana Cloud:

• Labels follow Grafana Cloud’s Linux Server integration (job="integrations/node_exporter", instance=<hostname>), so its dashboards work and the three hosts show up separately.
• prometheus-pve-exporter runs on pve-02 only. One exporter covers the whole cluster, because the Proxmox API returns every node, guest and storage. Alloy on pve-02 scrapes it.
• Secrets live in /etc/default/alloy and reach the config through sys.env, so the config files go into Git as they are.

The free tier has a series limit, and I hit it on the first day. The default exporter config sent about 7,100 series. Turning off the ZFS collector, skipping the VM tap interfaces and dm-* devices, and keeping only a few of Alloy’s own metrics brought it to about 2,500, plus about 670 from the PVE exporter.

Grafana Cloud is also where the host down and reboot alerts from Part 13 live: they have to run outside the cluster to catch a host taking the cluster down.

Result

kubectl -n monitoring exec vmsingle-0 -- wget -qO- http://127.0.0.1:8428/api/v1/series/count
# {"status":"success","data":[8493]}

About 1.9 GiB in total, against about 3 GiB for Coroot with ClickHouse. The biggest difference is per node: 170 MiB for the agent instead of 450 MiB, which matters on the 2 GiB pve-03 worker.

Lessons

1. Decide what you need before picking a tool. I wanted metrics, logs and alerts; a service map was nice to have and cost the most.
2. Prefer the standard for collection. With OTLP, the backends can change without touching the collectors or the apps.
3. Check what host metrics are labeled with. Metrics from a DaemonSet that carry no node label overwrite each other silently.
4. Changing provisioned identifiers needs a migration. For Grafana, deleteDatasources is that migration.
5. Know your free tier. Default exporters send far more series than a small lab needs; trim them before the limit does it for you.
6. Alerts about the platform belong outside it. The cluster alerts on pods; Grafana Cloud alerts on the hosts.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry (this post)
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google

References

• OpenTelemetry Collector Helm chart
• OpenTelemetry: Important components for Kubernetes
• VictoriaMetrics: OpenTelemetry ingestion
• VictoriaLogs: OpenTelemetry setup
• Grafana: Provision data sources
• Grafana community Helm charts
• prometheus-pve-exporter