Homelab Rebuild Part 37: A Power Trip, a Search Domain and a Wildcard

Intro

The power went out at home in the afternoon and came back about three hours later. The hosts booted, the VMs started, and kubectl hung. Nothing was broken. Every node was Ready, the API server answered, and Cilium (the cluster network) was healthy. Pods on one of the three hosts could not reach external hostnames. My kubeconfig goes through the Tailscale operator, which ran on that host. When that host lost the internet, I lost my way in.

TL;DR: Proxmox VMs with no searchdomain get the search domain from the host /etc/resolv.conf in their cloud-init data. After the power trip, the VMs autostarted about 50 seconds before tailscaled rewrote that file. So the pve-01 VMs booted with an old search franzramadhan.dev. Pods inherit the search list of the node and use ndots:5. The resolver first tried controlplane.tailscale.com as controlplane.tailscale.com.franzramadhan.dev, and a public wildcard *.franzramadhan.dev answered with the LAN IP of my Gateway. Every TLS connection to an external hostname from those pods got reset. The lasting fix is one line in Terraform: dns.domain = "franzramadhan.lab" on every VM.

The Figure

The figure is an addressograph, the machine that stamped addresses onto letters from a stack of metal plates. Each plate is one entry of the search list of the pod. The plates feed bottom first because controlplane.tailscale.com has fewer than five dots. A plate with no answer (NXDOMAIN) is blank. The press strikes, nothing prints, and the plate drops into the bin. The first plate with any answer inks the letter. With the knob on franzramadhan.lab, the node plate is blank. The bare name prints the real address, and the letter goes out. Switch it to franzramadhan.dev (click the machine, press L or D, or use the buttons). Then the wildcard embosses 192.168.101.152 on that plate. Every name stops there and goes to my own Gateway. Press 1, 2 or 3 to try api.github.com or releases.hashicorp.com. They end up in the same tray. I built it with the anatomy skill.

The Situation

• 3 Proxmox VE 9.2 hosts (pve-01, pve-02, pve-03), k0s on 8 Debian 13 VMs built from a cloud-init template (Parts 2 and 7): 3 controllers and 5 workers. In k0s, the controllers run the control plane (API server, etcd) but are not Kubernetes nodes. So kubectl get nodes lists only the 5 workers.
• VMs stop at 23:30 and start at 06:00 by a systemd timer. They also have onboot: 1, which makes Proxmox start them as soon as the host boots.
• Each host runs Tailscale, which manages its /etc/resolv.conf (nameserver 100.100.100.100, search <tailnet>.ts.net).
• My kubeconfig talks to the API server through the API proxy of the Tailscale operator (Part 4). Grafana, Headlamp, LiteLLM, OpenBao and Wazuh are reachable through proxy pods that the operator manages.
• A Cilium Gateway (Kubernetes Gateway API) serves my apps on the LAN IP 192.168.101.152. franzramadhan.dev has a public wildcard DNS record, *.franzramadhan.dev, that points to that IP.

The timeline, in WIB (UTC+7):

Issue 1: Healthy Cluster, No Way In

Symptom

kubectl get nodes hung. In tailscale status every in-cluster device (tailscale-operator, grafana, headlamp, litellm, openbao) was offline, “last seen 3h ago”. The hosts answered SSH, the Proxmox cluster had quorum (all 3 votes), and all 8 VMs were running.

Getting In Another Way

The API server address on the LAN answered:

ssh root@pve-01 'curl -sk -o /dev/null -w "%{http_code}\n" https://192.168.101.151:6443/readyz'
# 401

401 means the API server is up and refuses anonymous requests. A week earlier, my Mac could not reach the LAN from non-Apple binaries. So out of habit I tunnelled through pve-01 and pointed a copy of the LAN kubeconfig at the tunnel. Later that evening, the LAN kubeconfig worked directly from the Mac. I did not need the tunnel:

ssh -f -N -L 16443:192.168.101.151:6443 root@pve-01
sed 's#https://192.168.101.151:6443#https://127.0.0.1:16443#' kubeconfig > kc-lan
KUBECONFIG=kc-lan kubectl get nodes
# 5/5 Ready

All 5 worker nodes were Ready, with no stuck pods. The Cilium agent reported 72/72 healthy controllers and 5/5 nodes reachable. The cluster was fine. Only the way in was broken.

Cause, First Layer

Each in-cluster tailnet device is a proxy pod run by the Tailscale operator. The ones on k0s-pve-01-worker-2 had restarted 8 to 11 times:

health(warnable=not-in-map-poll): error: Unable to connect to the Tailscale coordination server
failed to get a reissued authkey: failed to receive new auth key: timeout waiting for auth key reissue after 10m0s

Their logs show the loop. The pod gets no connection to the Tailscale coordination server. Then it waits up to 10 minutes for the operator to issue a new auth key. Then it restarts. The operator, on the same node, was offline on the tailnet too. It logged only tsnet starting since its last restart at 19:50. The one proxy on pve-02 was fine. So this was about pods on pve-01, not about Tailscale itself.

Issue 2: The Node Had Internet, Its Pods Did Not

Symptom

From a proxy pod on a pve-01 worker, TLS to the coordination server failed. The same request from the node itself worked:

kubectl -n tailscale exec ts-headlamp-... -- wget -q -O /dev/null https://controlplane.tailscale.com/key
# ssl_client: SSL_connect
# wget: error getting response: Connection reset by peer

# on the node (VM 102)
curl -s -o /dev/null -w "%{http_code}\n" https://controlplane.tailscale.com/key
# 200

Dead End: Routing and NAT

I checked the obvious suspects for “node yes, pod no”:

• Tailscale routing table 52 (the routes Tailscale adds on each worker VM): only tailnet peer routes on both the pve-01 and pve-02 workers, no exit node, RouteAll false. Nothing there can catch internet traffic.
• Cilium masquerade (it rewrites the source IP of a pod to the IP of the node on the way out): the CILIUM_POST_nat MASQUERADE rule was there. cilium-dbg monitor --type drop showed no dropped IPv4 packets, only IPv6 and LLC noise.
• tcpdump on eth0 of the worker during a test from a pod showed a complete TLS session to 1.1.1.1, with the source NATed to the node IP.

That last test fooled me for a minute. The session was fine on the wire, but the pod reported a failure. https://1.1.1.1 redirects to a hostname, so my “IP only” test still depended on DNS. DNS was the real problem.

Cause

I resolved the name from Python in the LiteLLM pod, on the same node. That gave it away:

socket.getaddrinfo("controlplane.tailscale.com", 443)[0][4]
# ('192.168.101.152', 443)

That is my own Gateway (lab-gateway, a Cilium Gateway). Here is the /etc/resolv.conf of the headlamp proxy pod:

search tailscale.svc.cluster.local svc.cluster.local cluster.local franzramadhan.dev
nameserver 10.96.0.10
options ndots:5

Kubernetes gives pods ndots:5 by default, and the last search entry comes from the node. With ndots:5, the resolver treats any name with fewer than five dots as relative. It tries every search suffix before the bare name. The first suffix that gets an answer wins. NXDOMAIN (“no such name”) means it tries the next one. And franzramadhan.dev always answers:

dig +short '*.franzramadhan.dev' @1.1.1.1
# 192.168.101.152
dig +short controlplane.tailscale.com.franzramadhan.dev @1.1.1.1
# 192.168.101.152

So every external name, from controlplane.tailscale.com to api.github.com, resolved to the Gateway. The Gateway has no listener for those names, so it resets the handshake. I checked that from pve-01 with curl --resolve controlplane.tailscale.com:443:192.168.101.152: SSL_connect: Connection reset by peer.

On pve-02, the search domain of the node was franzramadhan.lab, which has no wildcard. Those lookups got NXDOMAIN and fell through to the real answer.

flowchart LR
  P[Pod asks for<br/>controlplane.tailscale.com] --> S{ndots:5<br/>try search list}
  S -->|cluster.local suffixes| N1[NXDOMAIN]
  S -->|pve-01 node:<br/>.franzramadhan.dev| W[public wildcard<br/>192.168.101.152]
  S -->|pve-02 node:<br/>.franzramadhan.lab| N2[NXDOMAIN]
  W --> G[Cilium Gateway<br/>TLS reset]
  N2 --> B[bare name<br/>real IP]

Issue 3: Where the Search Domain Came From

Symptom

Only the three VMs on pve-01 (two workers and controller-01) had the wrong domain:

# VM 101 (pve-01)
grep Domains /run/systemd/network/10-netplan-eth0.network
# Domains=franzramadhan.dev

# VM 201 (pve-02)
grep Domains /run/systemd/network/10-netplan-eth0.network
# Domains=franzramadhan.lab

It came from /etc/netplan/50-cloud-init.yaml. cloud-init rewrote that file at 19:48, during the boot after the power trip.

Cause

Proxmox hands each VM its cloud-init settings (IP, DNS, SSH keys) on a small virtual CD, the cloud-init drive, which it builds when the VM starts. My VMs had no searchdomain set. In that case, Proxmox fills the search domain from the /etc/resolv.conf of the host at that moment.

On a normal day, that file belongs to Tailscale. The VMs start at 06:00, long after tailscaled sets it. After the power trip, tailscaled was already running (19:48:34). But it writes /etc/resolv.conf only after it reaches its coordination server and gets a DNS configuration. At 19:49:31 it was still trying. At 19:49:37 it logged wgengine: Reconfig: configuring DNS. On pve-01, the VM started at 19:48:44. For those 53 seconds, the host still had its original file, which Tailscale keeps as a backup:

grep search /etc/resolv.pre-tailscale-backup.conf
# pve-01: search franzramadhan.dev
# pve-02: search franzramadhan.lab

The .dev there is most likely left from the install. The host name is still pve-01.franzramadhan.dev, while pve-02 is pve-02.franzramadhan.lab.

Inside the VM, cloud-init compares an instance-id from that drive with the one from the last boot. If the id is the same, it leaves the network configuration alone. If the id is new, it treats the boot as a new instance and writes the network configuration again. The drive that Proxmox built at 19:48:44 is still attached to the VM, so I could read exactly what it contained. The cloud-init log shows the id change on that boot:

# inside VM 101
mount -o ro /dev/sr0 /mnt && ls -l /mnt && grep -A1 search /mnt/network-config
# meta-data, network-config, user-data, vendor-data: all 12:48:44 UTC (19:48:44 WIB)
#       search:
#       - 'franzramadhan.dev'

grep 'previous iid' /var/log/cloud-init.log | tail -2
# 2026-10-20 12:48:55 ... previous iid found to be a2fbd92b...
# 2026-10-20 12:48:58 ... previous iid found to be d80d388d...

The old id, a2fbd92b..., was the same at the two previous 06:00 starts. Those boots logged No network config applied. Neither a new instance .... This boot got a new id, so cloud-init rewrote the netplan file from the new data, franzramadhan.dev included.

sequenceDiagram
  participant H as pve-01 host
  participant V as VM 101
  participant T as tailscaled
  T->>T: 19:48:34 started, no control server yet
  H->>V: 19:48:44 autostart, cloud-init drive built
  Note over H,V: host resolv.conf still says<br/>search franzramadhan.dev
  V->>V: cloud-init rewrites netplan<br/>Domains=franzramadhan.dev
  T->>H: 19:49:37 resolv.conf rewritten<br/>search <tailnet>.ts.net
  Note over H: too late for the VMs

One thing slowed me down here. qm cloudinit dump 101 network prints the cloud-init network data that Proxmox generates. It showed search: <tailnet>.ts.net, not .dev. The command reads the host resolv.conf as it is now, not as it was when the VM booted. So it looked fine while the VM was not.

Fix

First, the running nodes. I did not use netplan apply, because it can reconfigure interfaces. I did not want to risk a link change under Cilium on a live node. On each of the three pve-01 VMs, I ran:

sed -i s/franzramadhan.dev/franzramadhan.lab/ /etc/netplan/50-cloud-init.yaml
netplan generate                      # rewrites the networkd file, touches no link
resolvectl domain eth0 franzramadhan.lab

Pods copy the search list when they start. A fix on the node does not fix running pods. I recreated the pods that need external names. The Tailscale operator went first, then its five proxies. Then came LiteLLM, Grafana, the OpenTelemetry cluster collector, the GitHub Actions runner listener and one OpenBao pod (OpenBao unseals itself on start, so that is safe). The proxies were back on the tailnet within a minute, and kubectl through the usual kubeconfig worked again. I left alone the pods that only talk to in-cluster names or IPs (Postgres replica, VictoriaMetrics, VictoriaLogs, Pi-hole). An in-cluster name matches a cluster.local suffix before the resolver ever tries the bad one.

Then the real fix: pin the domain so it never depends on the host resolv.conf. The setting lives in Terraform. But a Terraform apply touches all three controllers in one run. If the provider reboots them together, etcd can lose quorum. So I set the value by hand first. That only edits the VM configuration and restarts nothing (it takes effect at the next VM start):

qm set 101 --searchdomain franzramadhan.lab   # and the other seven

and then made Terraform match:

dns {
  domain  = "franzramadhan.lab"
  servers = ["192.168.101.200", "1.1.1.3"]
}

The value was already set on every VM, so the plan said No changes. and the apply said Resources: 0 added, 0 changed, 0 destroyed. I also changed the pre-Tailscale backup file on pve-01 to franzramadhan.lab. The host no longer carries the old domain at all.

Results

The first CI runs after the fix still hit the problem once. The comment step of the Terraform plan failed with read tcp 10.0.2.129:41266->192.168.101.152:443: read: connection reset by peer. The first apply failed with read ECONNRESET in the setup-terraform step. The address in the first error is the same Gateway. I did not find out why a runner pod that started after the fix still went there. Both reruns passed, the apply with a few retried downloads.

Update 2026-10-21: The Same Bug Under .lab

The next afternoon, Flux stopped applying the Cilium Kustomization. It pulls the Gateway API CRDs from a GitHub release. The fetch failed with an address I knew by now:

Get "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml":
read tcp 10.0.3.249:43040->192.168.101.152:443: read: connection reset by peer

The controller pod already had the fixed search list (... franzramadhan.lab, ndots:5). So this was not a leftover from the power trip. From a test pod:

nslookup github.com.franzramadhan.lab
# Address: 192.168.101.152

The fix above swapped one wildcard for another. My Pi-hole answers every *.franzramadhan.lab name with the Gateway. That way the .lab names for my dashboards work without a DNS entry for each one. When I checked .lab as a search domain, I asked the router and Cloudflare. Both said NXDOMAIN. Pods do not ask either of them. CoreDNS forwards to the Tailscale resolver, 100.100.100.100, and the split DNS of the tailnet sends .lab to Pi-hole. So the resolver tried github.com as github.com.franzramadhan.lab first and got the Gateway. The TLS connection was reset. This also explains the CI runner in the results above that hit the Gateway after the fix.

It even blocked its own fix. Flux pulls this repo over SSH from github.com. That name resolved to the Gateway on port 22 too, so the commit with the fix could not reach the cluster. I suspended the two Pi-hole HelmReleases. I applied the same dnsmasq change to their ConfigMaps by hand and restarted the two Pi-holes one at a time. When Flux could fetch again, I resumed the releases.

The fix replaces the wildcard with one entry per routed name:

dnsmasq:
  customDnsEntries:
    - address=/grafana.franzramadhan.lab/192.168.101.152
    - address=/litellm.franzramadhan.lab/192.168.101.152
    # ... one line per HTTPRoute on the Gateway

github.com.franzramadhan.lab now returns NXDOMAIN from a pod. The dashboards still resolve, and the Cilium Kustomization applied on the next reconcile. The cost is one line in the Pi-hole values for every new .lab app.

Lessons

1. Unset is not “none”. A VM without a search domain does not get an empty one. It gets whatever the host has when the VM starts. Pin anything that must not depend on boot order.
2. Boot order is a dependency. The daily 06:00 start hid it every day so far. Only an autostart after a power loss raced tailscaled.
3. A public wildcard to a private IP turns small mistakes into outages. With ndots:5, any search domain that always answers hijacks every external name. That includes a private wildcard anywhere in the resolver chain. Test a search domain from a pod, not from the router.
4. Node fixed is not pod fixed. Pods copy resolv.conf at creation and keep it.
5. Make sure an “IP only” test really skips DNS. https://1.1.1.1 redirects to a hostname.
6. Keep a way in that skips the broken layer. The LAN kubeconfig does not go through Tailscale. It showed a healthy cluster within minutes and ruled out half the suspects. Test that path first. Do not assume that an old limitation still holds.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard (this post)
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• GitHub: wheresryan22/anatomy
• Proxmox VE: Cloud-Init Support
• Kubernetes: DNS for Services and Pods
• man7.org: resolv.conf(5)
• Tailscale: DNS in Tailscale
• Terraform Registry: bpg/proxmox virtual_environment_vm