Homelab Rebuild Part 13: The Host That Kept Losing Power

Intro

Part 12 ended with pve-01 back from a POST hang after a CMOS clear. That was not the end of it. Over the next ten hours the host reset more than 30 more times, first during busy moments, later every few minutes, and finally it stayed dark. Each time, the journal simply stopped mid-line.

This post is the whole hunt in one place: what the resets looked like, what I ruled out and how, and what fixed it.

TL;DR: A host whose journal stops mid-line with no shutdown sequence lost power. Disks, CMOS, kernel, thermals and VM load were all ruled out. Swapping the power adapters between pve-01 and pve-02 (same model) stopped the resets immediately. Host up/down and reboot alerts now run in Grafana Cloud, outside the cluster, because the in-cluster monitoring lives partly on the host that fails.

The Situation

• Three Proxmox VE 9 hosts: pve-01 and pve-02 are Lenovo ThinkCentre M715q Tiny (Ryzen, external power adapter), pve-03 is an old laptop
• pve-01 runs two k0s workers (VM 101, 102) and a controller (VM 200)
• One worker on pve-01 holds node-local data (/mnt/data) for CloudNativePG and the monitoring stack
• Part 12 already covered the first day: a reset loop in the morning of 2026-10-13, a POST hang until about 21:00, fixed by clearing the CMOS

Issue 1: Resets with no trace

Symptom

From pve-02, the pattern was always the same: corosync reports pve-01 leaving, then joining a minute later.

Oct 14 01:26:33 pve-02 corosync[1223]: [TOTEM ] A new membership (2.4f0) was formed. Members left: 1
Oct 14 01:27:10 pve-02 corosync[1223]: [TOTEM ] A new membership (1.4f5) was formed. Members joined: 1

On pve-01, journalctl --list-boots showed the scale of it. Every boot that day ended without a shutdown, and the last lines were ordinary noise, here Tailscale’s netcheck:

Oct 14 01:25:44 pve-01 tailscaled[912]: portmapper: PMP probe failed due result code: {OpCode:128 ResultCode:NetworkFailure ...}
Oct 14 01:25:44 pve-01 tailscaled[912]: netcheck: DetectCaptivePortal(found=false)
-- nothing more; the next entry is the kernel of the next boot --

The evening of 2026-10-13 was the worst of it:

• I sent poweroff once, and the host came back on and went off again about every 5 seconds.
• With the SSD attached, it booted into Proxmox, ran a filesystem check, then turned off again.
• On about the fifth try it stayed up, with two beeps and POST error 0162 (“Setup data integrity check failure”).
• VMs stored on the SSD failed to start with no such logical volume ssd/ssd on boots where the SSD was out (I had removed it to reach the CMOS battery).

The 0162 and the missing volume turned out to be side effects: 0162 is what the M715q reports on the first boot after a CMOS clear, and the volume was missing because the disk was physically out.

What it was not

A log that ends in the middle of normal output, with no panic and no shutdown, means the power went away. The question was why.

Dead end: it’s the load

The resets clustered around busy moments: early boot, all VMs starting at once, Helm rollouts. So I reduced the load. I stopped every VM that isn’t part of the cluster at 00:16 and disabled their autostart. In the cluster, I replaced Coroot (with its ClickHouse and an eBPF agent on every node) with a lighter OpenTelemetry stack (Part 14).

The longest stretch after that was 73 minutes. Then the resets came faster, even when the host was close to idle:

Load may have made it fail sooner, but it wasn’t the cause: only the k0s VMs were running for every reset on 2026-10-14. Shrinking uptimes with less and less load point at something physical getting worse.

Cause

Nothing in software could explain a host that loses power while idle. What was left on pve-01’s side: the board’s power circuitry, or the power input (the external adapter and its barrel plug).

Fix

The two M715q hosts use the same adapter, so the test was cheap: swap the adapters between pve-01 and pve-02 and see whether the fault moves.

At 06:49, pve-01 booted on pve-02’s adapter and pve-02 on pve-01’s. Both have been up since, with all k0s VMs running on both hosts. pve-01 hasn’t reset once.

The fault didn’t move to pve-02 so far, so I can’t yet say whether the old adapter is weak, or whether the plug or its contact was the problem and re-seating it fixed it. Either way it was the power input, not the board. If pve-02 starts resetting on its new adapter, the adapter is bad and gets replaced. The alerts below will tell me.

Issue 2: Finding out from the cluster is too late

Symptom

Every reset above I noticed by watching corosync from pve-02 by hand. The in-cluster monitoring (Part 14) can’t help much: Grafana, VictoriaMetrics and VictoriaLogs keep their data on a disk on k0s-pve-01-worker-1. When pve-01 loses power, the monitoring goes down with it.

Fix

The Proxmox hosts already ship metrics to Grafana Cloud through Grafana Alloy. That’s outside the cluster and outside the house, so the alerts go there. Two rules, in a rule group managed from a JSON file through the provisioning API:

The “down” rule doesn’t use up == 0. When a host loses power, Alloy stops sending, so up doesn’t become 0: the series just disappears. Comparing against the last day finds hosts that were there and aren’t anymore.

umask 077; printf 'Authorization: Bearer %s\n' "$(pbpaste)" > /tmp/gc.hdr
curl -s -H @/tmp/gc.hdr -H 'Content-Type: application/json' -H 'X-Disable-Provenance: true' \
  -X PUT https://<stack>.grafana.net/api/v1/provisioning/folder/<folder-uid>/rule-groups/pve-hosts \
  --data-binary @alerts/pve-hosts.json
rm /tmp/gc.hdr

Each rule sends email through a contact point named on the rule itself (notification_settings.receiver), so the stack’s default notification policy stays untouched.

One catch: a service account with No basic role gets 403 on everything. With Editor, it could create a folder but then couldn’t read it, so the rules went into an existing folder instead.

Lessons

1. A journal that stops mid-line means power. A panic, a hang or a shutdown leaves something behind. Nothing at all means the power went away.
2. Uptimes that keep shrinking point at hardware getting worse. Software faults tend to repeat at the same point. These came faster each time, even with less load.
3. Lower load can hide a power fault. It also kept me on the “too much load” theory for longer than it deserved.
4. Identical hardware gives you free swap tests. Disks (Part 12), then power adapters: each swap ruled something in or out in minutes.
5. Some POST errors are side effects. Error 0162 is expected after a CMOS clear; the missing LVM volume was a disk I had pulled myself.
6. Alert from outside what you’re watching. A monitoring stack with data on the failing host can’t report that host failing. The host alerts run in Grafana Cloud.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power (this post)
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google

References

• Lenovo: ThinkCentre M715q Tiny user guide
• systemd: journalctl –list-boots
• Grafana: Alerting provisioning HTTP API
• Grafana Alloy: prometheus.exporter.unix
• Proxmox VE: Cluster Manager