Homelab Rebuild Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go

Intro

In Part 13, pve-01 kept losing power until I replaced its adapter. The old adapter went to pve-02, and this morning pve-02 lost power. A new adapter is on the way.

I found out in the middle of the Headlamp SSO work (Part 20), when a command failed with “no route to host”. This post covers what a dead host meant for the cluster, the one service that couldn’t survive it, and a Postgres replica that needed a nudge to finish recovering.

TL;DR: With one of three hosts gone, etcd, Postgres and OpenBao kept working: two of three etcd members, a CloudNativePG failover, and two of three Raft voters. Pi-hole didn’t: its volume lives on one worker’s local disk, so the replacement pod had nowhere to run. After power came back, everything rejoined except the Postgres standby, which had replayed all WAL but waited for more. One pg_switch_wal() on the primary fixed it.

Update (same evening): pve-02 dropped twice more on the same adapter, and this time the logs showed how: the SATA SSD fails first, the host hangs, and only then does it drop. See the update at the end.

The Situation

• Three Proxmox hosts. pve-02 runs k0s-controller-02 and two workers, k0s-pve-02-worker-1 (with the NVMe data disk) and k0s-pve-02-worker-2
• CloudNativePG: cluster pg, one instance on each data-disk worker (pve-01 and pve-02)
• OpenBao: three Raft voters, one of them on pve-02
• Pi-hole: one replica, volume on local-path, so tied to one node; it’s the DNS server for the LAN and the tailnet
• pve-02 running on the adapter that came out of pve-01

Issue 1: pve-02 went dark

Symptom

error: unable to upgrade connection: error dialing backend:
dial tcp 192.168.101.51:10250: connect: no route to host

Both pve-02 workers were NotReady, controller-02’s lease had stopped renewing, and pve-01’s corosync log had the moment:

11:27:29 corosync[1359]: [KNET  ] host: host: 2 has no active links
11:27:30 corosync[1359]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms)
11:27:35 corosync[1359]: [TOTEM ] A new membership (1.5f8) was formed. Members left: 2

controller-02 had stopped renewing about 17 minutes earlier, at 11:10, so the host was in trouble before it dropped off the network completely.

Cause

Power. The adapter that failed on pve-01 in Part 13 is now failing on pve-02.

Fix

Reseat the adapter and the input cord. pve-02 was back at 11:34:35, Proxmox was quorate with three nodes again, and all five k0s nodes were Ready within 20 seconds. The real fix arrives tomorrow with the new adapter; until then pve-02 may drop again.

What kept running

This is what three of everything is for, and it held up.

Issue 2: Pi-hole had nowhere to go

Symptom

Kubernetes evicted the Pi-hole pod from the dead node and created a new one, which stayed Pending until pve-02 came back:

pihole-86bc89b448-54s2r   1/1   Terminating   k0s-pve-02-worker-1
pihole-86bc89b448-qzg65   0/1   Pending       <none>

Cause

Pi-hole keeps its settings on a local-path volume, and a local-path volume is a directory on one node’s disk. The pod can only run where its data is, and that node was off. With one replica, DNS for the LAN was down for the whole outage. Tailnet DNS has 1.1.1.3 as a fallback, so devices there should have kept resolving; LAN clients that only know Pi-hole didn’t.

Fix

For now, nothing beyond waiting: when pve-02 came back, the pending pod started on the same node and DNS returned. The lesson is the open item below.

Issue 3: A standby that replayed everything and still said no

Symptom

After the host came back, pg-2 stayed at 0/1 Ready for more than 11 minutes. Its log:

LOG  database system was interrupted while in recovery at log time 2026-10-15 03:14:05 UTC
LOG  restarting backup recovery with redo LSN 0/1D00CB00
LOG  entering standby mode
LOG  started streaming WAL from primary at 0/21000000 on timeline 5
FATAL the database system is not yet accepting connections   (1924 times)

But the primary said it was fully caught up:

pg-2|streaming|sent 0/21000000|replay 0/21000000|current 0/21000000

Cause

Most likely: the standby had replayed every WAL record the primary had, but hadn’t yet reached a consistent recovery point, so it wouldn’t accept connections. The primary was idle, sitting exactly on a segment boundary (0/21000000), and wasn’t writing any new WAL that would carry the standby past that point.

Fix

Make the primary write a little WAL:

SELECT pg_switch_wal();
CHECKPOINT;

Thirty seconds later:

pg-2|streaming|0/21000130|0/21000130
pg-1   1/1   Running   primary
pg-2   1/1   Running   replica
Cluster in healthy state ready=2/2

Result

All three hosts, five nodes, two Postgres instances and three OpenBao pods were back within minutes of power. The only user-visible outage was DNS on the LAN.

Lessons

1. Three of everything works. etcd, Postgres and OpenBao each lost a member and kept serving.
2. Local disks pin pods to hosts. Anything with one replica on local-path goes down with its host, and Pi-hole is exactly that. It needs a second instance, or a stateless setup with its config in Git.
3. An idle primary can leave a standby waiting. If a replica has replayed everything but still won’t accept connections, give it more WAL.
4. A suspect part stays suspect on another host. The adapter from Part 13 moved the problem from pve-01 to pve-02.
5. Power faults show up as something else. A sagging adapter looked like a failing SATA drive and a hung kernel. Compare when the journal stops with when the network goes; a long gap means it wasn’t a clean cut.

Open

• The new adapter arrives tomorrow.
• Wake-on-LAN is now armed on all three hosts (a small systemd unit sets wol g on each host’s uplink at boot, and Proxmox knows every node’s MAC), but it isn’t tested yet. It needs the BIOS settings and a real power-off test, which I’ll do in person.
• Pi-hole redundancy.

Update: two more drops, and what the logs show

pve-02 went down twice more on the same day, still on pve-01’s old adapter. pve-01 hasn’t dropped once since the swap, so the fault follows the adapter. I restarted pve-02 by pulling the power inlet for 20 seconds.

Symptom

journalctl --list-boots on pve-02 showed one outage nobody had noticed:

From 12:55 to 17:22 the host was off for four and a half hours, and from 18:26 to 18:50 it was down again.

A power cut that wasn’t instant

A clean power cut ends the journal and the network at the same moment (journald writes to disk every 5 minutes, so at most 5 minutes go missing). Here the journal stopped well before the network did:

That matches controller-02’s lease stopping at 11:10 in the original outage. The host hung first and dropped off later. There was no crash record: pstore was empty, with no machine-check or hardware-error lines.

The SATA SSD goes first

pve-02 has a Samsung 870 EVO 1TB on SATA (ata1) next to the NVMe boot drive. Before each death it degraded the same way:

ata1: limiting SATA link speed to 3.0 Gbps
ata1: limiting SATA link speed to 1.5 Gbps
ata1.00: limiting speed to UDMA/33:PIO4
ata1: illegal qc_active transition (00000000->00000001)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x100)
ata1.00: revalidation failed (errno=-5)
ata1: SATA link down (SStatus 0 SControl 310)
ata1.00: detaching (SCSI 0:0:0:0)

• Morning: errors from 11:05:57, drive detached 11:09:27, journal ends 11:10:14.
• Midday: errors from 11:40:47, detached 11:51:27; the host lasted until 12:55:47.
• Evening: errors from 18:21:45, 21 link resets, journal ends 18:26:24.

SMART on the drive says it’s healthy: 0 reallocated sectors, 0 uncorrectable errors, 0 CRC errors. The storage on it is empty on pve-02, so no VM lost data.

Cause (pending the new adapter)

My first reading was that the SATA path was an alternative cause. It isn’t: the adapter swap is stronger evidence. A worn adapter rarely cuts power cleanly. Under load its voltage sags, and the most sensitive link fails first. Here that’s the SATA SSD, powered through the board’s 5 V supply. The kernel hangs waiting for the drive, and finally the board browns out and drops to soft-off. The kernel’s AMD reset-reason line before the 17:22 boot said ACPI power state transition, not a reset.

The OS never logs the power problem itself, only its side effects: SATA errors, a hang, and a journal that stops early. On pve-01 the same adapter caused resets under load (Parts 12 and 13), and they stopped when the adapter moved.

Pulling the inlet was the right restart: after a brown-out the board can stay in a bad state until power is fully removed.

Next

The new adapter arrives tomorrow. If journalctl -k | grep ata1 stays quiet on pve-02 afterwards, the adapter was the cause. If the SATA errors continue, the drive, its cable or the port is next. I’ll update this post either way.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go (this post)
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox

References

• CloudNativePG: Failover
• PostgreSQL: Hot Standby
• PostgreSQL: pg_switch_wal
• Rancher local-path-provisioner
• Proxmox VE: Cluster Manager