Homelab Rebuild Part 12: Observability with Coroot, and Losing pve-01

| October 13, 2026 | 12 min to read

Intro

Update (2026-10-14): Coroot is gone. I replaced it with OpenTelemetry, VictoriaMetrics and VictoriaLogs a day later. See Part 14. The CMOS clear below was not the end for pve-01 either. It kept losing power until I swapped its power adapter. See Part 13.

After I hardened the cluster (Part 11), the next gap was visibility. I wanted to see what pods talk to, where latency comes from, and which workload eats the memory. I did not want to instrument every app. The Proxmox hosts already had Grafana Alloy installed for Grafana Cloud, so this post is only about the Kubernetes side.

The deploy itself went well. The data disk growth for it did not. A Terraform apply restarted two VMs, and the CI runner lost its Terraform state upload. Shortly after, the pve-01 host started resetting. Then it went dark for most of the day, until a CMOS clear brought it back.

TL;DR: Coroot 1.10.6 runs with VictoriaMetrics as its metrics store (through spec.externalPrometheus, so no Prometheus). A single-replica ClickHouse holds logs, traces and profiles. The Coroot operator deletes any Ingress named coroot, so give yours a different name. A disk resize through the bpg Proxmox provider restarts the VM, so drain the node first. pve-01 hung in POST until I cleared the CMOS.

The Situation

k0s cluster: 3 controllers, 5 workers across pve-01, pve-02 (Lenovo ThinkCentre M715q Tiny, Ryzen) and pve-03 (an old laptop whose battery lasts about 5 minutes)
Two workers (k0s-pve-01-worker-1, k0s-pve-02-worker-1) have an NVMe data disk at /mnt/data, labeled lab/data-disk=true, served by local-path
The pve-03 worker is tainted dedicated=quorum:NoSchedule and only 2 GiB
Cilium 1.20.2 and Tetragon 1.7.1 already run eBPF on every node
Terraform (bpg/proxmox) manages the Proxmox VMs. A self-hosted GitHub Actions runner that lives in the same cluster applies the changes

Choosing the Stack

I looked at lighter eBPF options first. One option is Grafana Beyla (now upstreamed as OpenTelemetry eBPF Instrumentation) inside Alloy, plus the Hubble metrics that Cilium already produces. That option ships everything to Grafana Cloud with no storage in the cluster. It is the lightest option.

I still wanted Coroot. The service map and its automatic inspections are the point. Most of the weight of Coroot is in its storage. The operator bundles Prometheus and ClickHouse by default. I did not want Prometheus, so I checked the CRD of coroot-operator 0.10.6:

# coroot-operator/files/crd.yaml
externalPrometheus:
  url:             ^https?://.+$
  remoteWriteURL:  ^https?://.+$
  basicAuth, customHeaders, tlsSkipVerify
storeMetricsInClickhouse: "Store metrics in ClickHouse. If enabled, Prometheus will not be installed."

I had two ways out. storeMetricsInClickhouse removes the metrics store. externalPrometheus points Coroot at anything that speaks the Prometheus query API and remote write. I picked VictoriaMetrics single, because it keeps a PromQL endpoint. Later I can point Grafana at it, or remote-write a subset to Grafana Cloud.

Component	Placement	Storage
`coroot-node-agent` (eBPF DaemonSet)	all 5 workers, including pve-03	none
`coroot-cluster-agent`	data-disk workers	none
Coroot server	data-disk workers	10Gi
ClickHouse (1 shard, 1 replica) + Keeper (1)	data-disk workers	100Gi + 10Gi
VictoriaMetrics single (30d)	data-disk workers	50Gi

Setup

Kustomize with two charts

The app follows the same layout as the rest of the cluster: one folder, with Helm charts pinned inside kustomization.yaml.

helmCharts:
  - name: coroot-operator
    repo: https://coroot.github.io/helm-charts
    releaseName: coroot-operator
    namespace: coroot
    version: 0.10.6
  - name: victoria-metrics-single
    repo: https://victoriametrics.github.io/helm-charts
    releaseName: vm
    namespace: coroot
    version: 0.49.0
    valuesFile: victoria-metrics-values.yaml

fullnameOverride: vmsingle gives VictoriaMetrics a stable Service name.

The Coroot object

apiVersion: coroot.com/v1
kind: Coroot
metadata:
  name: coroot
spec:
  authBootstrapAdminPasswordSecret:
    name: coroot-admin
    key: password
  externalPrometheus:
    url: http://vmsingle:8428
    remoteWriteURL: http://vmsingle:8428/api/v1/write
  nodeSelector:
    lab/data-disk: "true"
  clickhouse:
    shards: 1
    replicas: 1
    nodeSelector:
      lab/data-disk: "true"
    storage:
      className: local-path
      size: 100Gi
    keeper:
      replicas: 1
  nodeAgent:
    tolerations:
      - key: dedicated
        operator: Equal
        value: quorum
        effect: NoSchedule

The node-agent tolerates the quorum taint, so it covers pods on pve-03. Nothing stateful lands there. I generated the admin password into OpenBao at secret/coroot/admin, and External Secrets syncs it. My Kyverno policy from Part 11 allows this, because the path starts with the namespace name.

Pod Security

The node-agent needs privileged, hostPID and host paths. The operator deploys every component into the namespace of the Coroot object. So the coroot namespace sets pod-security.kubernetes.io/enforce: privileged explicitly. The Kyverno mutating policy that labels new namespaces baseline skips namespaces that already carry the label. The Pod Security Standards policies still report the node-agent, in Audit only.

Result

kubectl -n coroot exec vmsingle-0 -- wget -qO- http://127.0.0.1:8428/api/v1/series/count
# {"status":"success","data":[13435]}

The memory use on an idle lab: about 450 MiB per node for the node-agent (more than I expected), 500 MiB for ClickHouse, and 150 MiB each for Coroot and VictoriaMetrics.

Issue 1: The operator deleted my Ingress

Symptom

The Tailscale Ingress for the UI existed right after kubectl apply. Then it was gone. The LAN HTTPRoute kept working.

Cause

The operator log said it clearly:

INFO deleted {"namespace": "coroot", "name": "coroot", "type": "*v1.Ingress"}

The Coroot CRD has a spec.ingress field. The operator owns an Ingress named after the Coroot object. It deletes that Ingress when spec.ingress is empty. My Ingress had the same name by chance.

Fix

I renamed it to coroot-tailnet. Both endpoints returned 200 afterwards: https://coroot.<tailnet>.ts.net and https://coroot.franzramadhan.lab.

Issue 2: A disk resize that restarted VMs

Symptom

The data disks were 100G and the volumes already claimed about 180G. local-path does not enforce sizes, so it worked for a while, but it was going to stop. I changed data_disk = 100 to 250 in Terraform. The plan looked harmless:

~ disk {
    ~ size = 100 -> 250
  }
Plan: 0 to add, 2 to change, 0 to destroy.

The apply then failed after the change was complete:

proxmox_virtual_environment_vm.k0s["k0s-pve-01-worker-1"]: Modifications complete after 34s [id=101]
proxmox_virtual_environment_vm.k0s["k0s-pve-02-worker-1"]: Modifications complete after 45s [id=201]
Error: Failed to save state
... dial tcp: lookup default-terraform-state-....s3.ap-southeast-3.amazonaws.com
on 10.96.0.10:53: read udp ...: connection refused

Cause

“Updated in-place” in the plan did not mean “hot”. The Proxmox task log showed that the provider shut the VMs down and started them again:

UPID:pve-02:...:qmshutdown:201:terraform-apply@pve!github: OK

CoreDNS had pods on those two workers. The CI runner runs in the same cluster. When it tried to upload the new state to S3, it failed to resolve the bucket. Terraform wrote errored.tfstate locally. But the runner is an ephemeral pod, so that file disappeared with it.

Fix

I lost nothing that matters. The disks are 250G in Proxmox, and the next apply refreshes from Proxmox and saves the real size. Inside the guest, ext4 on a whole disk grows online:

DEV=$(findmnt -no SOURCE /mnt/data); echo 1 > /sys/class/block/${DEV#/dev/}/device/rescan; resize2fs $DEV
# /dev/sdb  246G  426M  235G   1% /mnt/data

Use findmnt, not a device name. After the restart, the data disk on one worker moved between sda and sdb.

Issue 3: pve-01 stopped coming back

Symptom

About 12 minutes after the apply, corosync on pve-02 started logging pve-01 leaving and rejoining every two to three minutes:

09:27:16 [KNET  ] link: host: 1 link: 0 is down
09:27:22 [TOTEM ] A new membership (2.472) was formed. Members left: 1
09:27:59 [TOTEM ] A new membership (1.477) was formed. Members joined: 1
09:29:32 [KNET  ] link: host: 1 link: 0 is down
...

It looked like the Realtek NIC from Part 10 again. It was not. I caught pve-01 in one of its up windows, and uptime said up 0 min. The whole host was resetting, and each boot started VM 101 again. Around 09:33 it stopped coming back at all. pve-02 showed its ARP entry as FAILED, so not even the NIC answered.

What it was not

Proxmox HA fencing (forced reset of a failed node). ha-manager status showed no HA resources and every watchdog in standby. Nothing was fencing pve-01.
The NIC fix from Part 10. I confirmed that EEE was off on that boot. The earlier boots that day ran for hours.
Anything I ran on the host. The only writes were to VM data disks through Proxmox. Nothing touched the boot disk of pve-01.

What brought it back

Someone at home did two remote power cycles. They did not help. When I got home in the evening, the disk LED was solid and never blinked. The machine was stuck in POST (the firmware self-test), before it read any disk. That also explains why pve-02 never saw its NIC.

First I wanted to know if the disks or the machine were at fault. So I swapped the disks with pve-02 (same model), with both other hosts powered off:

Test	Result
pve-01’s NVMe and SSD in pve-02’s chassis	boots normally
pve-02’s NVMe and SSD in pve-01’s chassis	hangs, no disk activity

The disks were fine. The problem stayed with the chassis of pve-01, so it was the board or its firmware state.

What worked, on the Lenovo M715q Tiny:

Unplug the power adapter, open the case (one screw at the back).
Remove both drives (the NVMe and the 2.5" SSD).
Take out the CMOS coin battery.
Move the jumper to pins 2 and 4, the position the board labels CLR CMOS.
Put everything back and power on.

It booted. The first boot started with the clock at 2026-04-14. That proved that the CMOS was cleared. NTP corrected the clock. Afterwards both drives looked healthy:

# NVMe: SMART overall-health PASSED, Percentage Used 4%, Media and Data Integrity Errors 0
# SSD:  SMART overall-health PASSED, Reallocated_Sector_Ct 0

The swap test ruled out the disks. So the most likely cause was a corrupted BIOS or CMOS state on pve-01 that left POST hanging. The reset loop in the morning possibly caused it. Afterwards I moved the jumper back to its normal position. Otherwise the CMOS clears on every power loss. The BIOS is now on defaults, so I must set “After Power Loss” to Power On again.

Recovery afterwards was uneventful. All VMs started, 5 of 5 nodes were Ready, etcd was back to 3 members and CloudNativePG was healthy with 2 instances. I also ran resize2fs on the second worker.

It did not last. Later that night pve-01 started resetting again. The cause was its power input, not the BIOS. That story is Part 13.

One more scare during recovery: the journal of pve-03 stopped at 21:17:01 with no shutdown entries. That was exactly like the resets of pve-01 in the morning. This time the cause was mundane. pve-03 is an old laptop whose battery lasts about 5 minutes, and its charger came loose. A log that stops tells you that power went away, not why.

A CI side effect: I pushed a commit while the cluster was down. Its Terraform Apply run stayed in the queue, because the in-cluster runner never received it. The next run waited behind it in the concurrency group of the workflow. It waited until I cancelled the stale one.

Lessons

The heavy part of observability is storage, not the agent. The choice of where metrics and logs live mattered more than which eBPF agent collects them.
Operators own names. Before you add your own object next to an operator, check if its CRD has a field that manages an object with the same kind and name.
“Update in-place” can still mean a restart. For the bpg provider, a disk resize restarts the VM. Drain the node first.
Do not let CI depend on what it changes. A runner inside the cluster lost DNS because of its own apply. If applies that touch node VMs run from outside the cluster, this does not happen.
Node-local storage is a single point of failure. With one host down, the data on its disk is unreachable until the host comes back.
A solid disk LED means POST, not the OS. No blinking at all meant the machine never got far enough to read a disk. So the fix had to be in firmware: a CMOS clear.
Identical hardware is a free diagnostic. A disk swap between two M715q units separated a disk problem from a board problem in minutes.
Check the power before suspecting the board. A log that stops with no shutdown entries only proves the power went away.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 12: Observability with Coroot, and Losing pve-01

Intro

The Situation

Choosing the Stack

Setup

Kustomize with two charts

The Coroot object

Pod Security

Result

Issue 1: The operator deleted my Ingress

Symptom

Cause

Fix

Issue 2: A disk resize that restarted VMs

Symptom

Cause

Fix

Issue 3: pve-01 stopped coming back

Symptom

What it was not

What brought it back

Lessons

Series

References