Homelab Rebuild Part 12: Observability with Coroot, and Losing pve-01

| October 13, 2026 | 10 min to read

Intro

After hardening the cluster (Part 11), the next gap was visibility. I wanted to see what pods talk to, where latency comes from, and which workload eats the memory, without instrumenting every app. The Proxmox hosts already had Grafana Alloy installed for Grafana Cloud, so this post is only about the Kubernetes side.

The deploy itself went well. Growing the data disks for it did not: a Terraform apply restarted two VMs, the CI runner lost its Terraform state upload, and shortly after, the pve-01 host started resetting and then went dark for most of the day, until a CMOS clear brought it back.

TL;DR: Coroot 1.10.6 runs with VictoriaMetrics as its metrics store (via spec.externalPrometheus, so no Prometheus) and a single-replica ClickHouse for logs, traces and profiles. The Coroot operator deletes any Ingress named coroot, so name yours differently. A disk resize through the bpg Proxmox provider restarts the VM, so drain the node first. pve-01 hung in POST until I cleared the CMOS.

The Situation

k0s cluster: 3 controllers, 5 workers across pve-01, pve-02 (Lenovo ThinkCentre M715q Tiny, Ryzen) and pve-03 (an old laptop whose battery lasts about 5 minutes)
Two workers (k0s-pve-01-worker-1, k0s-pve-02-worker-1) have an NVMe data disk at /mnt/data, labeled lab/data-disk=true, served by local-path
The pve-03 worker is tainted dedicated=quorum:NoSchedule and only 2 GiB
Cilium 1.20.2 and Tetragon 1.7.1 already run eBPF on every node
Proxmox VMs are managed by Terraform (bpg/proxmox), applied by a self-hosted GitHub Actions runner that lives in the same cluster

Choosing the Stack

I looked at lighter eBPF options first. Grafana Beyla (now upstreamed as OpenTelemetry eBPF Instrumentation) inside Alloy, plus Hubble metrics that Cilium already produces, would ship everything to Grafana Cloud with no storage in the cluster. That is the lightest option.

I still wanted Coroot: the service map and its automatic inspections are the point. What makes Coroot heavy is not its eBPF agent, it is the storage. The operator bundles Prometheus and ClickHouse by default. I didn’t want Prometheus, so I checked the CRD of coroot-operator 0.10.6:

# coroot-operator/files/crd.yaml
externalPrometheus:
  url:             ^https?://.+$
  remoteWriteURL:  ^https?://.+$
  basicAuth, customHeaders, tlsSkipVerify
storeMetricsInClickhouse: "Store metrics in ClickHouse. If enabled, Prometheus will not be installed."

Two ways out. storeMetricsInClickhouse removes the metrics store entirely. externalPrometheus points Coroot at anything that speaks the Prometheus query API and remote write. I picked VictoriaMetrics single, because it keeps a PromQL endpoint I can later point Grafana at, or remote-write a subset to Grafana Cloud.

Component	Placement	Storage
`coroot-node-agent` (eBPF DaemonSet)	all 5 workers, including pve-03	none
`coroot-cluster-agent`	data-disk workers	none
Coroot server	data-disk workers	10Gi
ClickHouse (1 shard, 1 replica) + Keeper (1)	data-disk workers	100Gi + 10Gi
VictoriaMetrics single (30d)	data-disk workers	50Gi

Setup

Kustomize with two charts

The app follows the same layout as the rest of the cluster: one folder, Helm charts pinned inside kustomization.yaml.

helmCharts:
  - name: coroot-operator
    repo: https://coroot.github.io/helm-charts
    releaseName: coroot-operator
    namespace: coroot
    version: 0.10.6
  - name: victoria-metrics-single
    repo: https://victoriametrics.github.io/helm-charts
    releaseName: vm
    namespace: coroot
    version: 0.49.0
    valuesFile: victoria-metrics-values.yaml

fullnameOverride: vmsingle gives VictoriaMetrics a stable Service name.

The Coroot object

apiVersion: coroot.com/v1
kind: Coroot
metadata:
  name: coroot
spec:
  authBootstrapAdminPasswordSecret:
    name: coroot-admin
    key: password
  externalPrometheus:
    url: http://vmsingle:8428
    remoteWriteURL: http://vmsingle:8428/api/v1/write
  nodeSelector:
    lab/data-disk: "true"
  clickhouse:
    shards: 1
    replicas: 1
    nodeSelector:
      lab/data-disk: "true"
    storage:
      className: local-path
      size: 100Gi
    keeper:
      replicas: 1
  nodeAgent:
    tolerations:
      - key: dedicated
        operator: Equal
        value: quorum
        effect: NoSchedule

The node-agent tolerates the quorum taint, so pods on pve-03 are covered, while nothing stateful lands there. The admin password is generated into OpenBao at secret/coroot/admin and synced by External Secrets, which my Kyverno policy from Part 11 allows because the path starts with the namespace name.

Pod Security

The node-agent needs privileged, hostPID and host paths, and the operator deploys every component into the namespace of the Coroot object. So the coroot namespace sets pod-security.kubernetes.io/enforce: privileged explicitly. The Kyverno mutating policy that labels new namespaces baseline skips namespaces that already carry the label. The Pod Security Standards policies still report the node-agent, in Audit only.

Result

kubectl -n coroot exec vmsingle-0 -- wget -qO- http://127.0.0.1:8428/api/v1/series/count
# {"status":"success","data":[13435]}

Footprint on an idle lab: about 450 MiB per node for the node-agent (more than I expected), 500 MiB for ClickHouse, 150 MiB each for Coroot and VictoriaMetrics.

Issue 1: The operator deleted my Ingress

Symptom

The Tailscale Ingress for the UI existed right after kubectl apply, then it was gone. The LAN HTTPRoute kept working.

Cause

The operator log said it plainly:

INFO deleted {"namespace": "coroot", "name": "coroot", "type": "*v1.Ingress"}

The Coroot CRD has a spec.ingress field. The operator owns an Ingress named after the Coroot object and deletes it when spec.ingress is empty. Mine happened to have the same name.

Fix

Rename it to coroot-tailnet. Both endpoints returned 200 afterwards: https://coroot.<tailnet>.ts.net and https://coroot.franzramadhan.lab.

Issue 2: A disk resize that restarted VMs

Symptom

The data disks were 100G and the volumes already claimed about 180G (local-path doesn’t enforce sizes, so it worked, until it wouldn’t). I changed data_disk = 100 to 250 in Terraform. The plan was harmless:

~ disk {
    ~ size = 100 -> 250
  }
Plan: 0 to add, 2 to change, 0 to destroy.

The apply then failed after the change was done:

proxmox_virtual_environment_vm.k0s["k0s-pve-01-worker-1"]: Modifications complete after 34s [id=101]
proxmox_virtual_environment_vm.k0s["k0s-pve-02-worker-1"]: Modifications complete after 45s [id=201]
Error: Failed to save state
... dial tcp: lookup default-terraform-state-....s3.ap-southeast-3.amazonaws.com
on 10.96.0.10:53: read udp ...: connection refused

Cause

“Updated in-place” in the plan did not mean “hot”. The Proxmox task log showed the provider shut the VMs down and started them again:

UPID:pve-02:...:qmshutdown:201:terraform-apply@pve!github: OK

CoreDNS had pods on those two workers. The CI runner runs in the same cluster, so when it tried to upload the new state to S3, it could not resolve the bucket. Terraform wrote errored.tfstate locally, but the runner is an ephemeral pod, so that file disappeared with it.

Fix

Nothing was lost that matters: the disks are 250G in Proxmox, and the next apply refreshes from Proxmox and saves the real size. Inside the guest, ext4 on a whole disk grows online:

DEV=$(findmnt -no SOURCE /mnt/data); echo 1 > /sys/class/block/${DEV#/dev/}/device/rescan; resize2fs $DEV
# /dev/sdb  246G  426M  235G   1% /mnt/data

Use findmnt rather than a device name: after the restart, the data disk on one worker had moved between sda and sdb.

Issue 3: pve-01 stopped coming back

Symptom

About 12 minutes after the apply, corosync on pve-02 started logging pve-01 leaving and rejoining every two to three minutes:

09:27:16 [KNET  ] link: host: 1 link: 0 is down
09:27:22 [TOTEM ] A new membership (2.472) was formed. Members left: 1
09:27:59 [TOTEM ] A new membership (1.477) was formed. Members joined: 1
09:29:32 [KNET  ] link: host: 1 link: 0 is down
...

It looked like the Realtek NIC from Part 10 again. It wasn’t: when I caught pve-01 in one of its up windows, uptime said up 0 min. The whole host was resetting, and each boot started VM 101 again. Around 09:33 it stopped coming back at all. pve-02 showed its ARP entry as FAILED, so not even the NIC was answering.

What it was not

Proxmox HA fencing. ha-manager status showed no HA resources and every watchdog in standby. Nothing was fencing pve-01.
The NIC fix from Part 10. EEE was confirmed off on that boot, and the earlier boots that day had run for hours.
Anything I ran on the host. The only writes were to VM data disks through Proxmox; pve-01’s own boot disk was never touched.

What brought it back

Two remote power cycles (by someone at home) did not help. When I got home in the evening, the disk LED was solid and never blinked: the machine was stuck in POST, before it read any disk. That also explains why pve-02 never saw its NIC.

First I wanted to know whether the disks or the machine were at fault, so I swapped them with pve-02 (same model), with both other hosts powered off:

Test	Result
pve-01’s NVMe and SSD in pve-02’s chassis	boots normally
pve-02’s NVMe and SSD in pve-01’s chassis	hangs, no disk activity

The disks were fine. The problem moved with pve-01’s chassis, so it was the board or its firmware state.

What worked, on the Lenovo M715q Tiny:

Unplug the power adapter, open the case (one screw at the back).
Remove both drives (the NVMe and the 2.5" SSD).
Take out the CMOS coin battery.
Move the jumper to pins 2 and 4, the position the board labels CLR CMOS.
Put everything back and power on.

It booted. The first boot started with the clock at 2026-04-14, proof that the CMOS really was cleared, and NTP corrected it. Afterwards both drives looked healthy:

# NVMe: SMART overall-health PASSED, Percentage Used 4%, Media and Data Integrity Errors 0
# SSD:  SMART overall-health PASSED, Reallocated_Sector_Ct 0

With the swap test ruling out the disks, the most likely cause was corrupted BIOS/CMOS state on pve-01 that left POST hanging, possibly left behind by the reset loop in the morning. Afterwards I moved the jumper back to its normal position, otherwise the CMOS would clear on every power loss. The BIOS is now on defaults, so “After Power Loss” needs to be set to Power On again.

Recovery afterwards was uneventful: all VMs started, 5 of 5 nodes Ready, etcd back to 3 members, CloudNativePG healthy with 2 instances, and resize2fs on the second worker.

One more scare during recovery: pve-03’s journal stopped at 21:17:01 with no shutdown entries, exactly like pve-01’s resets in the morning. This time it was mundane. pve-03 is an old laptop whose battery lasts about 5 minutes, and its charger had come loose. A log that just stops tells you power went away, not why.

A CI side effect: the Terraform Apply run for a commit pushed while the cluster was down stayed queued, because the in-cluster runner never received it. The next run waited behind it in the workflow’s concurrency group until I cancelled the stale one.

Lessons

The heavy part of observability is storage, not the agent. Choosing where metrics and logs live mattered more than which eBPF agent collects them.
Operators own names. Before adding your own object next to an operator, check whether its CRD has a field that manages an object with the same kind and name.
“Update in-place” can still mean a restart. For the bpg provider, a disk resize restarts the VM. Drain the node first.
Don’t let CI depend on what it changes. A runner inside the cluster lost DNS because of its own apply. Running applies that touch node VMs from outside the cluster avoids this.
Node-local storage is a single point of failure. With one host down, the data on its disk is unreachable until the host comes back.
A solid disk LED means POST, not the OS. No blinking at all meant the machine never got far enough to read a disk. Clearing the CMOS was the fix, not anything on the Linux side.
Identical hardware is a free diagnostic. Swapping disks between two M715q units separated a disk problem from a board problem in minutes.
Check the power before suspecting the board. A log that stops with no shutdown entries only proves the power went away.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 12: Observability with Coroot, and Losing pve-01

Intro

The Situation

Choosing the Stack

Setup

Kustomize with two charts

The Coroot object

Pod Security

Result

Issue 1: The operator deleted my Ingress

Symptom

Cause

Fix

Issue 2: A disk resize that restarted VMs

Symptom

Cause

Fix

Issue 3: pve-01 stopped coming back

Symptom

What it was not

What brought it back

Lessons

Series

References