Back to Home
post-thumb

Homelab Rebuild Part 27: One Bad Power Adapter, Two Different Deaths

Technology Homelab
| October 17, 2026 | 10 min to read

Intro

For four days my two ThinkCentre hosts kept dying. pve-01 went first (Parts 12 and 13 ): resets every few minutes, a POST hang, then silence. Swapping its power adapter with pve-02’s stopped it. A day later pve-02, now on pve-01’s old adapter, started dying too (Part 21 ), but in a completely different way.

This post puts the whole thing in one place: what each host did, how the logs pointed at the adapter and not at the disks, and what the new adapter shows so far.

TL;DR: One worn adapter, two failure modes. On pve-01 it caused instant resets with zero disk errors. On pve-02 it caused a slow brown-out: the SATA SSD degraded and detached, the host hung for 14 to 52 minutes, then dropped. pve-02’s SSD had 0 errors in 34 boots over 16 months on its own adapter and errors in every boot on the bad one. On a new adapter it has logged 0 errors so far (about 16 hours on 2026-10-17 morning). Not final yet: on the bad adapter the first error took 27 hours.

The Figure

CH1 01 · CH2 02ABNABN01026.03.01.5016.03.01.502
SATA link speed lampPatch cord: which brick feeds which PCBrick A is the worn adapter. On pve-01 (01) it drops the rail to zero at once: a hard reset, the journal stops mid-line, and the SSD logs no errors. S swaps A and B. On pve-02 (02) the rail sags instead: the SATA link steps from 6.0 to 3.0 to 1.5 Gbps, goes down and detaches, the host hangs for 14 to 52 minutes, then drops to soft-off. On its own brick B, pve-02 logged 0 ata1 errors in 34 boots; on A, errors in every boot (41, 7, 10, 5 and 15). N puts the new brick on pve-02: steady, 0 ata1 errors so far (about 16 hours; on A the first error took 27). 1 and 2 pick the host the readout follows, R goes back to the start, a click steps through the three setups.

The figure is a test bench. Bricks A, B and N on the left are the power adapters: A is the worn one, B is pve-02’s own, N is the new one. Patch cords on the panel decide which brick feeds which mini PC, 01 (pve-01) or 02 (pve-02). Each PC has its SATA SSD on top and three lamps on the front for the link speed, and the scope at the back draws one supply trace per PC. It starts with A on pve-01: the trace drops to zero at once and the SSD lamp never moves. S swaps A and B: on pve-02 the trace sags, the link steps down from 6.0 to 3.0 to 1.5 Gbps, then goes dark. N puts the new brick on pve-02, 1 and 2 pick the host the readout follows, and R starts over. I generated it with the anatomy agent skill.

The Situation

  • pve-01 and pve-02: Lenovo ThinkCentre M715q Tiny (Ryzen 5 2400GE), each on its own external power adapter (same model).
  • Storage in each: an NVMe drive (Proxmox, VM disks) and a Samsung 870 EVO 1TB SATA SSD (ata1).
  • Proxmox VE 9 cluster with a third host (pve-03, a laptop) for quorum.
  • Alloy on every host ships kernel logs to Grafana Cloud, outside the cluster.
  • Timeline:
    • 2026-10-13 and 10-14: pve-01 resets, POST hang, power-loss loop.
    • 2026-10-14 06:49: adapters swapped between pve-01 and pve-02.
    • 2026-10-15: pve-02 dies five times.
    • 2026-10-16 16:57: pve-02 on a new adapter.

Issue 1: pve-01 died instantly

Symptom

On pve-01’s old adapter, the journal stopped mid-line, with no shutdown sequence and nothing from the kernel. The 10-14 morning was a loop of boots lasting 0 to 1 seconds. Once the board hung at POST until the CMOS was cleared.

What the disks said

Nothing. I searched every crash boot from 10-13 and 10-14 (77 boots in the journal back to 10-12) for SATA trouble:

for b in $(journalctl --list-boots --no-pager | awk '{print $1}'); do
  printf '%s ' "$b"
  journalctl -k -b "$b" --no-pager \
    | grep -cE 'exception|hard resetting|limiting|IDENTIFY|detaching|reset failed'
done

Zero error lines in every crash boot. ata1: SATA link down appeared only in three boots, while the SSD was unplugged on purpose to reach the CMOS battery. The SSD on pve-01 never complained about the bad adapter.

Issue 2: pve-02 died slowly

Symptom

pve-02 ran 27 hours on pve-01’s old adapter without an error. Then, on 10-15, it died five times. Before each death the SATA SSD degraded in the same order:

ata1: limiting SATA link speed to 3.0 Gbps
ata1: limiting SATA link speed to 1.5 Gbps
ata1.00: limiting speed to UDMA/33:PIO4
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x100)
ata1: SATA link down (SStatus 0 SControl 310)
ata1.00: detaching (SCSI 0:0:0:0)

Then the host hung, and only much later did the cluster notice it was gone.

Hang, then drop

journald writes to disk every 5 minutes, so a clean power cut loses at most 5 minutes of journal. Here, the journal ended long before corosync on the other hosts lost pve-02:

DeathSSD detachedJournal endsCorosync loses pve-02Gap
1 (nobody noticed)10:00:2010:07:2010:58:2651 min
211:09:2711:10:1411:27:2917 min
311:51:2712:55:4713:48:0752 min
4errors from 18:21:4518:26:2418:40:3714 min
519:22:0919:43:2920:02:2819 min

The first one only showed up later in journalctl --list-boots: the host had reset itself at 10:58:55, after 51 minutes of hanging, and rejoined before anyone looked.

There was no crash record. pstore was empty, and there were no machine-check lines. The AMD reset-reason register, which the kernel prints at boot, said the board went down through a soft-off, not a reset:

x86/amd: Previous system reset reason [0x00200800]: ACPI power state transition occurred

SMART on the SSD said it was healthy: 0 reallocated sectors, 0 uncorrectable errors, 0 CRC errors. One number stood out, though: 20 unexpected power losses in 48 power cycles.

Was it the SSD?

That was my first reading, and it’s the obvious one: the SSD fails first every time. The SSD’s own history says otherwise. pve-02’s journal keeps every boot back to June 2025, all with the same drive:

BootsPeriodAdapterata1 error lines
34 boots2025-06-09 to 2026-10-14 06:49its own0
1 boot2026-10-14 06:52 to 10-15 10:07pve-01’s old one41
4 boots2026-10-15pve-01’s old one7, 10, 5 and 15

Sixteen months clean, then errors in every single boot from the day the adapter arrived. The disk didn’t change. The power did.

Cause

A worn adapter doesn’t always cut out cleanly. Under load its voltage can sag. On pve-02 the most sensitive part was the SATA SSD, so it failed first. The kernel then hung waiting for the drive, and finally the board browned out to soft-off. On pve-01 the same adapter caused hard resets instead. Why one board resets and the other sags, I can’t tell from the logs: it could be load, board tolerance or the drive. I won’t claim more than the logs show.

The red herring: pve-01’s SSD failed on a good adapter

On 10-16, after pve-02 got its new adapter, pve-01 (on pve-02’s original, good adapter) booted clean at 16:57. Three minutes later, during a VM start, its own 870 EVO threw hundreds of errors:

ata1.00: exception Emask 0x0 SAct 0x400002ff SErr 0x0 action 0x6 frozen
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata1: limiting SATA link speed to 3.0 Gbps

By 17:15 every read returned DID_BAD_TARGET and the storage on that drive was gone. Unlike pve-02, the host itself stayed up. I drained its Kubernetes workers, shut it down and power-cycled it without reseating anything. It came back at 6.0 Gbps with SMART passing, and the drive has been clean since.

That adapter had run pve-01 for 31 hours without a single error, so this event doesn’t fit the adapter story. It could be a one-off drive or controller hang, the drive itself, or its connection: I had unplugged that SSD twice for CMOS clears on 10-13. It’s on the watch list, not in the verdict.

Fix

  1. A new adapter for pve-02 on 2026-10-16. The worn one is out of use.

  2. An alert before the drop. A Grafana Cloud rule watches the hosts' kernel logs for SATA and NVMe trouble. On pve-02 these errors came 15 to 50 minutes before each drop, which is time to act:

    count_over_time({job="integrations/node_exporter"}
  |~ `ata[0-9.]+: (failed to IDENTIFY|reset failed|limiting SATA link speed|hard resetting link)|nvme[0-9]+: (I/O|controller is down|Abort|timeout)|I/O error, dev` [10m]) > 0

    I removed SATA link down from the pattern: pve-03 logs it at every boot for an empty port.

  3. A gate on the morning start. pve-02’s VMs only come back at 06:00 when the disk log is clean. A one-shot timer counts ata1 errors before it turns the start timer back on. It counted 0 and turned it on.

Verdict, observed so far

Checked on 2026-10-17 at 09:08 WIB, per boot:

for b in -1 0; do journalctl -k -b $b | grep -ciE 'ata1.*(fail|error|link down|detach)'; done
HostAdapterSinceata1 errorsLink
pve-02new10-16 16:57 (2 boots, about 16 h)06.0 Gbps, UDMA/133
pve-01pve-02’s original10-16 17:46 (about 15 h)06.0 Gbps, UDMA/133

pve-02 rebooted once in that window, on purpose, during an unrelated LAN problem (Part 25 ).

So far everything points at the adapter. It isn’t proven yet. On the bad adapter, the first error came after about 27 hours, so the new one has to pass that mark, around 2026-10-17 20:00 WIB, and then keep going. If an error shows up, the alert fires, and the drive, its cable or the port is next. I’ll add the result here either way.

Lessons

  1. Compare when the journal ends with when the network goes. A gap longer than journald’s sync interval means a hang, not a power cut.
  2. Keep the boot history. 34 clean boots on the same disk is what turned “the SSD is dying” into “the power is bad”.
  3. The first component to fail isn’t always the cause. The SATA SSD was the most sensitive part, not the broken one.
  4. Swap one variable. Moving the adapter moved the fault. Moving nothing else is what made that convincing.
  5. Look for the outage nobody noticed. journalctl --list-boots found a death that had already healed itself.
  6. Leave room for the event that doesn’t fit. pve-01’s SSD failure on a good adapter is still open, and calling it part of the adapter story would have been easier and wrong.
  7. Say “so far” until the evidence covers the time it took to fail.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths (this post)

References