Homelab Rebuild Part 25: It Was the Router

| October 16, 2026 | 10 min to read

Intro

On Friday evening pve-02 fell off the network. Tailscale could not reach it, and corosync lost its links. From the console, the host looked healthy: the interface was up, the link LED blinked, and it could ping its own IP.

Three days earlier, in Part 10, pve-01’s Realtek NIC stalled in exactly this way. All three hosts have the same chip, so I went straight for the NIC. I was wrong, and I rebooted a host for nothing before I found out.

TL;DR: The NICs were fine. Their TX and RX counters kept moving, while a ping to the gateway took 5.5 seconds from every host. The cause was the TP-Link AX5400 Wi-Fi router, which is also the only switch in my LAN. A router restart brought the gateway RTT back to 0.5 ms. Check the counters twice and ping the gateway from two hosts before you blame a driver.

The Figure

Lamps: a line's port LED. Lever: restartTickets: calls waiting for an answerJ jams the board, 1 to 6 unplug a line, R pulls the restart lever, S adds the planned gigabit switch. The board is the TP-Link AX5400: gateway, Wi-Fi and the only LAN switch, so every call between two lines is patched through it. Lines: 01 pve-01, 02 pve-02, 03 pve-03, 04 internet, 05 Wi-Fi, 06 CCTV. Jammed, every lamp blinks fast, the meter reads 5571 ms to the gateway, pve-01 to pve-02 loses every packet, and tickets pile up. Unplugging one line changes nothing, as the bisect showed. The restart clears the queue and the meter drops back to about 0.5 ms. With the switch, the three hosts reach each other without the board, and a jam only costs internet.

The figure is an operator’s telephone switchboard, because that is what the router is in my LAN. It patches through every call between two lines. Lines 01 to 03 are the Proxmox hosts, 04 is the internet trunk, 05 is Wi-Fi and 06 is the CCTV camera. The meter on the face reads the gateway RTT on a log scale, and the tickets on the shelf are calls waiting for an answer. Press J to jam the board: every lamp blinks fast, the meter climbs to 5571 ms, and the tickets pile up. 1 to 6 unplug a line. That changes nothing, like my bisect. R pulls the restart lever. S adds the gigabit switch that I plan to buy, so the hosts keep talking while the board is jammed. I generated it with the anatomy agent skill.

The Situation

Three Proxmox VE 9 hosts: pve-01 and pve-02 (Lenovo ThinkCentre Tiny), pve-03 (an old laptop). All three have a Realtek RTL8168/8111 NIC on the in-kernel r8169 driver.
Since Part 10, every host runs with EEE off and pcie_aspm=off.
No separate switch. A TP-Link AX5400 Wi-Fi 6 router is the gateway (192.168.101.200), the Wi-Fi access point, and the LAN switch. Its four ports go to pve-01, pve-02, pve-03 and a CCTV camera.
Corosync, Kubernetes and node-to-node Tailscale traffic all cross that router’s built-in switch.

flowchart LR
  NET((Internet)) --- R[TP-Link AX5400<br/>gateway .200<br/>switch + Wi-Fi]
  R --- P1[pve-01]
  R --- P2[pve-02]
  R --- P3[pve-03]
  R --- CAM[CCTV]
  R -.- WIFI[Wi-Fi clients]

Issue 1: It looked like the NIC stall again

Symptom

From the pve-02 console:

vmbr0 and enp1s0f0 were UP,LOWER_UP, and the port LED was blinking.
ping 192.168.101.202 (its own IP) answered. Pings to anything else timed out.
ip neigh listed a lot of STALE entries.
tailscaled logged network is unreachable and netcheck: UDP is blocked, trying HTTPS, and corosync showed both peers' links down.

That matched Part 10 almost line for line. So I checked whether the Part 10 mitigations were actually in place:

grep -o pcie_aspm=off /proc/cmdline
# pcie_aspm=off
ethtool --show-eee enp1s0f0
# EEE status: disabled
journalctl -k -b | grep -iE 'r8169|watchdog|timeout'
# only the normal probe and link up lines, no tx timeout

Dead end: two checks that prove less than they seem

A ping to your own IP never touches the NIC. The kernel answers it over loopback, so it proves only that the IP stack is alive.
A blinking link LED means only that the PHY has a link. In Part 10 the link stayed up while the NIC passed nothing.

Dead end: rebooting pve-02

I rebooted pve-02 from the console at 21:08. It came back with working neighbors, and that seemed to confirm the NIC theory. Eight minutes later, corosync on pve-02 logged that pve-01 went away:

corosync[1269]: [KNET  ] link: host: 1 link: 0 is down
corosync[1269]: [KNET  ] host: host: 1 has no active links

Now a second host had the same symptom, and both already ran the Part 10 fix. A shared cause was much more likely than two NIC failures in the same evening.

Issue 2: The NICs were fine, the LAN was not

Symptom

On the pve-01 console I ran ip -s link show enp1s0f0 twice, a little while apart:

Counter	First	Second
RX packets	4458688	4486705
TX packets	3830449	3862741
TX errors	0	0

A stalled NIC does not move its TX counter. This one moved by about 32,000 packets. Then the gateway:

ping -c3 192.168.101.200
# 64 bytes from 192.168.101.200: icmp_seq=1 ttl=64 time=5490 ms
# 64 bytes from 192.168.101.200: icmp_seq=2 ttl=64 time=5570 ms
# 64 bytes from 192.168.101.200: icmp_seq=3 ttl=64 time=5654 ms
# rtt min/avg/max/mdev = 5489.797/5571.384/5654.099/67.080 ms
ping -c3 192.168.101.202
# 3 packets transmitted, 0 received, 100% packet loss

There was no loss to the gateway, but each reply took 5.5 seconds. On a LAN, a reply normally takes well under a millisecond. From pve-02, pve-03 answered in 21 ms. Every path was slow, and every path went through the router.

On the way, I also tripped over my own gateway address. I pinged 192.168.101.1 out of habit and got FAILED in ip neigh. ip route says the gateway is 192.168.101.200, so that entry was meaningless.

Cause

The LAN port LEDs on the router all blinked fast, and I plugged in nothing new. I unplugged the hosts one at a time. Every remaining machine and VM still saw multi-second latency. No single wired host was flooding the LAN. The one device left that every packet crossed was the router itself.

I did not capture the CPU, uptime or client list of the router before I restarted it. So I cannot say whether it hung or a Wi-Fi client flooded it. To unplug wired ports does not rule out a wireless client.

Fix

I restarted the router. Afterwards, from all three hosts:

ping -c3 -q 192.168.101.200
# pve-01: rtt min/avg/max/mdev = 0.419/0.516/0.604/0.075 ms
# pve-02: rtt min/avg/max/mdev = 0.306/0.441/0.572/0.108 ms
# pve-03: rtt min/avg/max/mdev = 0.390/0.483/0.551/0.068 ms
pvecm status | grep -E 'Nodes|Quorate'
# Nodes:            3
# Quorate:          Yes

	Before	After
Gateway RTT (avg)	5571 ms	about 0.5 ms
pve-01 to pve-02	100% loss	0% loss
Corosync	2 of 3 nodes	3 of 3, quorate

Side effect of the reboot

pve-02 had a one-shot systemd-run timer from Part 21. At 23:00 it was set to re-enable the morning VM start, if the disk stayed clean. Transient timers do not survive a reboot, so I re-armed it. This time it counts disk errors across both boots with journalctl -k --since.

To do

A dedicated gigabit switch for the three hosts. Today the router is a single point of failure for the cluster’s own traffic, and also for internet access. The plan puts the hosts on their own switch with one uplink to the router. Then a router hang costs internet access, but not corosync quorum or node-to-node traffic. The CCTV camera stays on the router. Done on 2026-10-17, see the update below.
```
flowchart LR
  NET((Internet)) --- R[TP-Link AX5400]
  R --- CAM[CCTV]
  R -.- WIFI[Wi-Fi clients]
  R --- SW[Gigabit switch]
  SW --- P1[pve-01]
  SW --- P2[pve-02]
  SW --- P3[pve-03]
```
Next time, look at the router before you restart it. Uptime, CPU and the connected client list can tell a hang apart from a Wi-Fi client that floods the LAN.

Update 2026-10-17: the switch is in

I bought a TP-Link TL-SG108, an unmanaged 8-port gigabit desktop switch, sooner than planned. I installed it the evening of 2026-10-17. I measured it the next morning. The topology is now exactly the to-do diagram: ISP modem, then the AX5400, then the switch with one uplink, then the three hosts. The CCTV camera stays on a router port.

Every host NIC came up at 1000Mb/s, full duplex, with 0 RX and TX errors:

ethtool enp1s0f0 | grep -e Speed -e Duplex
# Speed: 1000Mb/s
# Duplex: Full

I ran iperf3 between every pair of hosts, 10 s each, on port 60001. The Proxmox host firewall opens only its own ports, and 60000-60050 is the live migration range:

Path	Throughput	Retransmits
pve-01 to pve-02, and back	939 Mbit/s each way	0
pve-01 to pve-03, and back	939 / 940 Mbit/s	0
pve-02 to pve-03, and back	940 Mbit/s each way	0
pve-01 and pve-02, both at once	931 + 928 Mbit/s	0

That is line rate for gigabit Ethernet. Gateway RTT from the hosts is 0.53 to 0.58 ms, the same as after the router restart. The router is still a single point of failure for internet access and the tailnet. But a hang like the one in this post no longer cuts corosync, etcd or Cilium traffic between the hosts.

Lessons

Check the counters twice before you blame the driver. A stalled NIC freezes its TX counter. A moving counter clears the NIC in ten seconds.
Ping the gateway from two hosts. Multi-second RTT on several hosts at once is a shared device, not a host problem.
A ping to your own IP and a blinking LED prove almost nothing. One is loopback, and the other is only the PHY.
Know your real gateway. ip route first, then test the address it shows.
A recent incident biases the diagnosis. The Part 10 stall was fresh, so I matched the symptoms to it. I rebooted a host before I checked the one thing that all hosts share.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 25: It Was the Router

Intro

The Figure

The Situation

Issue 1: It looked like the NIC stall again

Symptom

Dead end: two checks that prove less than they seem

Dead end: rebooting pve-02

Issue 2: The NICs were fine, the LAN was not

Symptom

Cause

Fix

Side effect of the reboot

To do

Update 2026-10-17: the switch is in

Lessons

Series

References