Homelab Rebuild Part 13: The Host That Kept Losing Power

Intro

Part 12 ended with pve-01 back from a POST hang after a CMOS clear. That was not the end of it. Over the next ten hours the host reset more than 30 more times. First the resets came during busy moments, later every few minutes. Finally the host stayed dark. Each time, the journal stopped mid-line.

This post is the whole hunt in one place. It covers what the resets looked like, what I ruled out and how, and what fixed it.

TL;DR: A host whose journal stops mid-line with no shutdown sequence lost power. I ruled out disks, CMOS, kernel, thermals and VM load. I swapped the power adapters between pve-01 and pve-02 (same model), and the resets stopped immediately. Host up/down and reboot alerts now run in Grafana Cloud, outside the cluster. The reason: part of the in-cluster monitoring lives on the host that fails.

The Situation

• Three Proxmox VE 9 hosts: pve-01 and pve-02 are Lenovo ThinkCentre M715q Tiny (Ryzen, external power adapter), pve-03 is an old laptop
• pve-01 runs two k0s workers (VM 101, 102) and a controller (VM 200)
• One worker on pve-01 holds node-local data (/mnt/data) for CloudNativePG and the monitoring stack
• Part 12 covered the first day: a reset loop in the morning of 2026-10-13, then a POST hang until about 21:00. Clearing the CMOS fixed the hang

Issue 1: Resets with no trace

Symptom

From pve-02, the pattern was always the same. Corosync reports that pve-01 left, then that it joined again a minute later.

Oct 14 01:26:33 pve-02 corosync[1223]: [TOTEM ] A new membership (2.4f0) was formed. Members left: 1
Oct 14 01:27:10 pve-02 corosync[1223]: [TOTEM ] A new membership (1.4f5) was formed. Members joined: 1

On pve-01, journalctl --list-boots showed the scale of it. Every boot that day ended without a shutdown. The last lines were ordinary noise, in this case the Tailscale netcheck:

Oct 14 01:25:44 pve-01 tailscaled[912]: portmapper: PMP probe failed due result code: {OpCode:128 ResultCode:NetworkFailure ...}
Oct 14 01:25:44 pve-01 tailscaled[912]: netcheck: DetectCaptivePortal(found=false)
-- nothing more; the next entry is the kernel of the next boot --

The evening of 2026-10-13 was the worst of it:

• I sent poweroff once. After that, the host came back on and went off again about every 5 seconds.
• With the SSD attached, it booted into Proxmox and ran a filesystem check. Then it turned off again.
• On about the fifth try it stayed up, with two beeps and POST error 0162 (“Setup data integrity check failure”).
• On boots where the SSD was out, VMs stored on the SSD failed to start with no such logical volume ssd/ssd. I removed the SSD to reach the CMOS battery.

The 0162 and the missing volume were side effects. The M715q reports 0162 on the first boot after a CMOS clear. The volume was missing because the disk was physically out.

What it was not

A log that ends in the middle of normal output, with no panic and no shutdown, means that the power went away. The question was why.

Dead end: it is the load

The resets clustered around busy moments: early boot, all VMs starting at once, and Helm rollouts. So I reduced the load. At 00:16 I stopped every VM that is not part of the cluster and disabled their autostart. In the cluster, I replaced Coroot with a lighter OpenTelemetry stack (Part 14). Coroot ran ClickHouse and an eBPF agent on every node.

The longest stretch after that was 73 minutes. Then the resets came faster, even when the host was close to idle:

Load possibly made it fail sooner, but it was not the cause. Only the k0s VMs were running for every reset on 2026-10-14. Uptimes got shorter while the load got lower. That points at something physical that gets worse.

Cause

Nothing in software can explain a host that loses power while idle. Two suspects were left on pve-01: the power circuitry on the board, or the power input (the external adapter and its barrel plug).

Fix

The two M715q hosts use the same adapter, so the test was cheap. I swapped the adapters between pve-01 and pve-02 to see whether the fault moves.

At 06:49, pve-01 booted on the adapter from pve-02, and pve-02 booted on the one from pve-01. Both stayed up after that, with all k0s VMs running on both hosts. pve-01 did not reset once.

So far the fault did not move to pve-02. So I cannot yet say which of two things was the problem. The old adapter is weak, or the plug contact was bad and re-seating it fixed it. Either way it was the power input, not the board. If pve-02 starts to reset on its new adapter, the adapter is bad, and I replace it. The alerts below will tell me.

Issue 2: Finding out from the cluster is too late

Symptom

I noticed every reset above because I watched corosync from pve-02 by hand. The in-cluster monitoring (Part 14) cannot help much. Grafana, VictoriaMetrics and VictoriaLogs keep their data on a disk on k0s-pve-01-worker-1. When pve-01 loses power, the monitoring goes down with it.

Fix

The Proxmox hosts already ship metrics to Grafana Cloud through Grafana Alloy. Grafana Cloud is outside the cluster and outside the house, so the alerts go there. I made two rules in one rule group. A JSON file manages the group through the provisioning API:

The “down” rule does not use up == 0. When a host loses power, Alloy stops sending, so up does not become 0. The series disappears instead. A comparison against the last day finds hosts that were there and are not there anymore.

umask 077; printf 'Authorization: Bearer %s\n' "$(pbpaste)" > /tmp/gc.hdr
curl -s -H @/tmp/gc.hdr -H 'Content-Type: application/json' -H 'X-Disable-Provenance: true' \
  -X PUT https://<stack>.grafana.net/api/v1/provisioning/folder/<folder-uid>/rule-groups/pve-hosts \
  --data-binary @alerts/pve-hosts.json
rm /tmp/gc.hdr

Each rule sends email through a contact point that the rule names itself (notification_settings.receiver). So the default notification policy of the stack stays untouched.

One catch: a service account with No basic role gets 403 on everything. With Editor, it could create a folder but then could not read it. So I put the rules into an existing folder instead.

Lessons

1. A journal that stops mid-line means power. A panic, a hang or a shutdown leaves something behind. Nothing at all means that the power went away.
2. Uptimes that keep shrinking point at hardware that gets worse. Software faults usually repeat at the same point. These came faster each time, even with less load.
3. Lower load can hide a power fault. It also kept me on the “too much load” theory for longer than it deserved.
4. Identical hardware gives you free swap tests. First disks (Part 12), then power adapters. Each swap ruled something in or out in minutes.
5. Some POST errors are side effects. Error 0162 is normal after a CMOS clear. The missing LVM volume was a disk that I pulled myself.
6. Alert from outside the thing you watch. A monitoring stack with data on the failing host cannot report that the host fails. The host alerts run in Grafana Cloud.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power (this post)
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• Lenovo: ThinkCentre M715q Tiny user guide
• systemd: journalctl –list-boots
• Grafana: Alerting provisioning HTTP API
• Grafana Alloy: prometheus.exporter.unix
• Proxmox VE: Cluster Manager