Homelab Rebuild Part 24: Pi-hole Without a Home Host

| October 16, 2026 | 7 min to read

Intro

In Part 21 pve-02 lost power, and Pi-hole went down with it. Its settings sat on a local-path volume on k0s-pve-02-worker-1, so the pod could only run there. pve-02 dropped four more times the next day, and each time the LAN lost DNS.

I wanted Pi-hole to survive any one host going away. My first idea was to move its state to the Postgres cluster, which already fails over between hosts. Pi-hole can’t do that (it only speaks SQLite), but it turned out it didn’t need to.

TL;DR: Pi-hole had almost no state worth keeping. Everything that matters (local DNS, upstreams, the blocklist) now lives in Helm values, so I run two replicas without a volume, one per Proxmox host. The upgrade failed first, because a field owned by the old Helm CLI survived server-side apply. One upgrade with serverSideApply: disabled fixed that. Then both pods landed on one host, until I added matchLabelKeys: [pod-template-hash] to the spread rule. Deleting the pod that held the DNS IP lost 0 of 40 queries.

The Situation

Pi-hole 2026.07.2, chart pihole 2.38.0, managed by Flux (helm-controller v1.6.5).
One replica, 1 GiB local-path PVC, pinned to the data-disk workers.
LAN DNS on 192.168.101.161, a LoadBalancer IP announced by Cilium L2 with externalTrafficPolicy: Local, so the query log shows real client IPs.
The same pods also serve the tailnet through a Tailscale LoadBalancer.
Workers: two on pve-01, two on pve-02, one on pve-03. The pve-03 worker is tainted for quorum only. Nodes carry topology.kubernetes.io/zone set to their host.

What was actually on the volume

Before deciding anything, I looked at what Pi-hole stored:

kubectl -n pihole exec deploy/pihole -- pihole-FTL sqlite3 /etc/pihole/gravity.db \
  "select address from adlist; select count(*) from domainlist;"
# https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
# 0

One default blocklist, no allow or deny entries. The *.franzramadhan.lab wildcard and the Proxmox host names were already in the Helm values as dnsmasq lines. The only thing the volume really held was the query log and its stats.

So the fix was not “make the state highly available”. It was “stop having state”: put the blocklist in values too, drop the volume, and run two copies.

replicaCount: 2

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app: pihole
        release: pihole

podDisruptionBudget:
  enabled: true
  minAvailable: 1

adlists:
  - https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

persistentVolumeClaim:
  enabled: false

nodeTaintsPolicy: Honor leaves tainted nodes out of the spread. That keeps the quorum-only pve-03 worker out, and lets both pods share a host while the other host is down.

Issue 1: The upgrade rejected its own Deployment

Symptom

Flux applied the change and the HelmRelease failed:

Helm upgrade failed for release pihole/pihole with chart pihole@2.38.0:
server-side apply failed for object pihole/pihole apps/v1, Kind=Deployment:
Deployment.apps "pihole" is invalid:
[spec.template.spec.volumes[0].persistentVolumeClaim: Forbidden: may not specify more than 1 volume type,
 spec.template.spec.containers[0].volumeMounts[0].name: Not found: "config"]

The old pod kept running, so DNS stayed up.

Cause

The new chart output had volume config as emptyDir: {}. The live Deployment had config as persistentVolumeClaim. Server-side apply merged the two into one volume with both sources, and the API refused it.

It didn’t remove the old field because the old field wasn’t fully Flux’s. The managed fields showed two owners:

helm             Apply   persistentVolumeClaim
helm-controller  Apply   persistentVolumeClaim

I installed Pi-hole with the Helm CLI before Flux took over (Part 16 ). That install is the helm manager, and it still owned the field. Server-side apply only removes a field when no other manager owns it, so helm-controller dropping it from its own config wasn’t enough.

Dead end: `upgrade.force`

The usual advice is force: true. The field description says why that does nothing here:

Force forces resource updates through a replacement strategy
that avoids 3-way merge conflicts on client-side apply.
This field is ignored for server-side apply

Fix

One upgrade with client-side apply:

spec:
  upgrade:
    serverSideApply: disabled

A client-side 3-way merge compares the previous release (which had the PVC) with the new one (which doesn’t) and sends a patch that deletes the PVC source. The upgrade went through, and Helm removed the old PVC itself. I took the setting out again in the next change.

Issue 2: Both pods on the same host

Symptom

pihole-6d65594f98-6spz4   1/1   Running   k0s-pve-01-worker-1
pihole-6d65594f98-t486s   1/1   Running   k0s-pve-01-worker-1

Two replicas, one node, one host. Exactly what the spread rule should prevent.

Cause

During the rolling update, the old pod was still running on pve-02. It has the same app and release labels, so the scheduler counted it: pve-02 already had one Pi-hole, pve-01 had none, and both new pods went to pve-01 without breaking maxSkew: 1. Then the old pod was removed.

Fix

Count only pods from the same ReplicaSet:

    matchLabelKeys:
      - pod-template-hash

The next rollout put one pod on each host:

pihole-796b7b55c4-8hxws   1/1   Running   k0s-pve-02-worker-1
pihole-796b7b55c4-clwf2   1/1   Running   k0s-pve-01-worker-1

Failover test

Cilium announces 192.168.101.161 from one node with a local Pi-hole pod (the holder of the cilium-l2announce-pihole-pihole-dns lease). I deleted the pod on the holder while pve-03 queried the IP once per second:

for i in $(seq 1 40); do
  dig +short +time=1 +tries=1 @192.168.101.161 example.com | head -1
  sleep 1
done

	Value
Lease holder before	`k0s-pve-01-worker-1`
Lease holder after	`k0s-pve-02-worker-1`
Failed queries	0 of 40
Replacement pod	`k0s-pve-01-worker-2`

Result

LAN DNS on the same IP, now served from both hosts.
No volume, so nothing ties Pi-hole to a node anymore.
A PodDisruptionBudget keeps one replica up during drains.
Trade-off: each replica has its own query log and stats, both lost on restart, and the web UI shows one replica per page load.

Lessons

Check what the state is before making it highly available. I was ready to fight Pi-hole into Postgres. A two-line SQL check showed there was nothing to move.
Helm CLI history lives on in managed fields. Flux adopted the release, but the helm field manager stayed. Removing a value can fail months later on a field Flux never fully owned.
upgrade.force is a client-side apply option. Under server-side apply it is ignored. serverSideApply: disabled for one upgrade is the switch that matters.
Topology spread counts every matching pod, old ones too. Without matchLabelKeys: [pod-template-hash], a rolling update can stack the new pods away from the old one.
Test failover by removing the thing that holds the IP. Deleting a random pod proves little. Deleting the one behind the L2 lease proves the IP moves.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 24: Pi-hole Without a Home Host

Intro

The Situation

What was actually on the volume

Issue 1: The upgrade rejected its own Deployment

Symptom

Cause

Dead end: upgrade.force

Fix

Issue 2: Both pods on the same host

Symptom

Cause

Fix

Failover test

Result

Lessons

Series

References

Dead end: `upgrade.force`