Homelab Rebuild Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go

Intro

In Part 13, pve-01 kept losing power until I replaced its adapter. The old adapter went to pve-02, and this morning pve-02 lost power. A new adapter is on the way.

I found out in the middle of the Headlamp SSO work (Part 20), when a command failed with “no route to host”. This post covers what a dead host meant for the cluster and the one service that did not survive it. It also covers a Postgres replica that needed a nudge to finish recovery.

TL;DR: With one of three hosts gone, etcd, Postgres and OpenBao kept working: two of three etcd members, a CloudNativePG failover, and two of three Raft voters. Pi-hole did not. Its volume lives on one worker’s local disk, so the replacement pod had nowhere to run. After power came back, everything rejoined except the Postgres standby. It replayed all WAL but waited for more. One pg_switch_wal() on the primary fixed it.

Update (same evening): pve-02 dropped twice more on the same adapter. This time the logs showed how: the SATA SSD fails first, the host hangs, and only then does it drop. See the update at the end.

The Situation

• Three Proxmox hosts. pve-02 runs k0s-controller-02 and two workers, k0s-pve-02-worker-1 (with the NVMe data disk) and k0s-pve-02-worker-2
• CloudNativePG: cluster pg, one instance on each data-disk worker (pve-01 and pve-02)
• OpenBao: three Raft voters, one of them on pve-02
• Pi-hole: one replica, volume on local-path, so tied to one node. It is the DNS server for the LAN and the tailnet
• pve-02 running on the adapter that came out of pve-01

Issue 1: pve-02 went dark

Symptom

error: unable to upgrade connection: error dialing backend:
dial tcp 192.168.101.51:10250: connect: no route to host

Both pve-02 workers were NotReady, and controller-02’s lease no longer renewed. The corosync log on pve-01 showed the moment:

11:27:29 corosync[1359]: [KNET  ] host: host: 2 has no active links
11:27:30 corosync[1359]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms)
11:27:35 corosync[1359]: [TOTEM ] A new membership (1.5f8) was formed. Members left: 2

controller-02 stopped renewing about 17 minutes earlier, at 11:10. So the host was in trouble before it dropped off the network completely.

Cause

Power. The adapter that failed on pve-01 in Part 13 is now failing on pve-02.

Fix

Reseat the adapter and the input cord. pve-02 was back at 11:34:35, and Proxmox was quorate with three nodes again. All five k0s nodes were Ready within 20 seconds. The real fix arrives tomorrow with the new adapter. Until then, pve-02 can drop again.

What kept running

This is what three of everything is for, and it held up.

Issue 2: Pi-hole had nowhere to go

Symptom

Kubernetes evicted the Pi-hole pod from the dead node and created a new one. The new pod stayed Pending until pve-02 came back:

pihole-86bc89b448-54s2r   1/1   Terminating   k0s-pve-02-worker-1
pihole-86bc89b448-qzg65   0/1   Pending       <none>

Cause

Pi-hole keeps its settings on a local-path volume, and a local-path volume is a directory on one node’s disk. The pod can run only where its data is, and that node was off. With one replica, DNS for the LAN was down for the whole outage. Tailnet DNS has 1.1.1.3 as a fallback, so devices there probably kept resolving. LAN clients that know only Pi-hole did not.

Fix

For now, nothing beyond waiting. When pve-02 came back, the pending pod started on the same node and DNS returned. The lesson is the open item below.

Issue 3: A standby that replayed everything and still said no

Symptom

After the host came back, pg-2 stayed at 0/1 Ready for more than 11 minutes. Its log:

LOG  database system was interrupted while in recovery at log time 2026-10-15 03:14:05 UTC
LOG  restarting backup recovery with redo LSN 0/1D00CB00
LOG  entering standby mode
LOG  started streaming WAL from primary at 0/21000000 on timeline 5
FATAL the database system is not yet accepting connections   (1924 times)

But the primary said it was fully caught up:

pg-2|streaming|sent 0/21000000|replay 0/21000000|current 0/21000000

Cause

Most likely, the standby replayed every WAL record the primary had. But it did not yet reach a consistent recovery point (a WAL position where the data is safe to read), so it did not accept connections. The primary was idle and sat exactly on a segment boundary (0/21000000). It wrote no new WAL to carry the standby past that point.

Fix

Make the primary write a little WAL:

SELECT pg_switch_wal();
CHECKPOINT;

Thirty seconds later:

pg-2|streaming|0/21000130|0/21000130
pg-1   1/1   Running   primary
pg-2   1/1   Running   replica
Cluster in healthy state ready=2/2

Result

All three hosts, five nodes, two Postgres instances and three OpenBao pods were back within minutes of power. The only user-visible outage was DNS on the LAN.

Lessons

1. Three of everything works. etcd, Postgres and OpenBao each lost a member and kept serving.
2. Local disks pin pods to hosts. Anything with one replica on local-path goes down with its host, and Pi-hole is exactly that. It needs a second instance, or a stateless setup with its configuration in Git.
3. An idle primary can leave a standby waiting. If a replica replayed everything but does not accept connections, give it more WAL.
4. A suspect part stays suspect on another host. The adapter from Part 13 moved the problem from pve-01 to pve-02.
5. Power faults show up as something else. A sagging adapter looked like a failing SATA drive and a hung kernel. Compare when the journal stops with when the network goes. A long gap means it was not a clean cut.

Open

• The new adapter arrives tomorrow.
• Wake-on-LAN is now armed on all three hosts. A small systemd unit sets wol g on each host’s uplink at boot, and Proxmox knows every node’s MAC. I did not test it yet. It needs the BIOS settings and a real power-off test, which I will do in person.
• Pi-hole redundancy.

Update: two more drops, and what the logs show

pve-02 went down twice more on the same day, still on pve-01’s old adapter. pve-01 did not drop once since the swap, so the fault follows the adapter. I restarted pve-02 by pulling the power inlet for 20 seconds.

Symptom

journalctl --list-boots on pve-02 showed one outage that nobody noticed:

From 12:55 to 17:22 the host was off for four and a half hours, and from 18:26 to 18:50 it was down again.

A power cut that was not instant

A clean power cut ends the journal and the network at the same moment. journald writes to disk every 5 minutes, so at most 5 minutes go missing. Here the journal stopped well before the network did:

That matches controller-02’s lease stopping at 11:10 in the original outage. The host hung first and dropped off later. There was no crash record. pstore was empty, with no machine-check lines or hardware-error lines.

The SATA SSD goes first

pve-02 has a Samsung 870 EVO 1TB on SATA (ata1) next to the NVMe boot drive. Before each death it degraded the same way:

ata1: limiting SATA link speed to 3.0 Gbps
ata1: limiting SATA link speed to 1.5 Gbps
ata1.00: limiting speed to UDMA/33:PIO4
ata1: illegal qc_active transition (00000000->00000001)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x100)
ata1.00: revalidation failed (errno=-5)
ata1: SATA link down (SStatus 0 SControl 310)
ata1.00: detaching (SCSI 0:0:0:0)

• Morning: errors from 11:05:57, drive detached 11:09:27, journal ends 11:10:14.
• Midday: errors from 11:40:47, detached 11:51:27. The host lasted until 12:55:47.
• Evening: errors from 18:21:45, 21 link resets, journal ends 18:26:24.

SMART on the drive says it is healthy: 0 reallocated sectors, 0 uncorrectable errors, 0 CRC errors. The storage on it is empty on pve-02, so no VM lost data.

Cause (pending the new adapter)

My first reading was that the SATA path was an alternative cause. It is not. The adapter swap is stronger evidence. A worn adapter rarely cuts power cleanly. Under load its voltage sags, and the most sensitive link fails first. Here that is the SATA SSD, powered through the board’s 5 V supply. The kernel hangs while it waits for the drive. Finally the board browns out (its voltage drops too low) and goes to soft-off. The kernel’s AMD reset-reason line before the 17:22 boot said ACPI power state transition, not a reset.

The OS never logs the power problem itself, only its side effects: SATA errors, a hang, and a journal that stops early. On pve-01 the same adapter caused resets under load (Parts 12 and 13), and they stopped when the adapter moved.

Pulling the inlet was the right restart. After a brown-out, the board can stay in a bad state until you fully remove power.

Next

The new adapter arrives tomorrow. If journalctl -k | grep ata1 stays quiet on pve-02 afterwards, the adapter was the cause. If the SATA errors continue, I will look at the drive, its cable or the port next. I will update this post either way.

Update: the adapter verdict, so far (2026-10-17)

pve-02 ran on a new adapter from 2026-10-16 16:57. In about 16 hours and two boots it logged 0 ata1 errors, at 6.0 Gbps and UDMA/133. Its SSD also had 0 errors in 34 boots over 16 months on its original adapter. The fault followed the old adapter, not the drive. The full root cause, with pve-01’s different failure mode and one SSD event that does not fit, is in Part 27.

Update: Pi-hole has two homes now (2026-10-16)

The “Pi-hole redundancy” open item is done. There are two replicas without a volume, one per host, and a failover test lost no queries. The details, including two upgrade surprises, are in Part 24. The adapter verdict comes in a later post.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go (this post)
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• CloudNativePG: Failover
• PostgreSQL: Hot Standby
• PostgreSQL: pg_switch_wal
• Rancher local-path-provisioner
• Proxmox VE: Cluster Manager