Intro
I wanted every service in the rebuilt cluster to have a name and a certificate
that my devices trust. That applies to the public domain I own
(franzramadhan.dev). It also applies to internal names that will never be
public (franzramadhan.lab, which is not a registered domain).
TL;DR: cert-manager issues a Let’s Encrypt wildcard for
*.franzramadhan.devthrough Cloudflare DNS-01 and runs a private root CA for*.franzramadhan.lab, trusted on every node. Pi-hole resolves the.labzone for the LAN and, through a Tailscale LoadBalancer, for every tailnet device. external-dns writes Cloudflare records, but only with the newexternal-dns.kubernetes.io/annotation prefix.
The Figure
The two presses are the two issuers of cert-manager. The left one carries the
public Let’s Encrypt die, fed by the Cloudflare token box, with a selector that
stamps on staging before production. The right one carries the house die, wired
to the lab-root-ca block on its stand. The cabinet is Pi-hole, with one lamp
for a LAN query and one for a tailnet query. The flag on the out box is
external-dns, which writes to Cloudflare. Press 1 or 2 (or click a press) to
stamp a card. Press S to switch staging and production, 3 for a DNS lookup
and 4 to flip the annotation prefix. I built the figure with the
anatomy agent skill.
The Situation
- cert-manager v1.21.2 with Gateway API support
- Pi-hole 2026.07.2 (chart 2.38.0)
- external-dns 0.23.0 (chart 1.23.0)
lab-gateway(Cilium) on192.168.101.152with HTTPS listeners- The Cloudflare API token lives in OpenBao (Part 5)
Setup
Public: a Let’s Encrypt wildcard
DNS-01 (proof of domain control through a DNS TXT record) is the only way to get a wildcard. It also works for a cluster that the internet cannot reach. The token needs only Zone DNS Edit and Zone Read on the one zone. I issued from the staging endpoint first, so a mistake does not burn production rate limits:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-franzramadhan-dev
namespace: gateway
spec:
secretName: wildcard-franzramadhan-dev
dnsNames:
- "*.franzramadhan.dev"
issuerRef:
kind: ClusterIssuer
name: letsencrypt
Staging issued in about 90 seconds. Production issued right after.
Internal: a private root CA
cert-manager can bootstrap a CA. A self-signed issuer signs one root
certificate, and a CA ClusterIssuer uses it for everything else.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: lab-root-ca
namespace: cert-manager
spec:
isCA: true
commonName: franzramadhan.lab Root CA
secretName: lab-root-ca
duration: 87600h # 10 years
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
kind: ClusterIssuer
name: selfsigned
The root certificate goes into the trust store of each node
(/usr/local/share/ca-certificates/ plus update-ca-certificates). It also goes
into the cloud-init snippet for future VMs and into the keychain on my Mac. The OpenBao
listener and the *.franzramadhan.lab gateway listener both use certificates
from this CA.
Both listeners share port 443 on the same Gateway, and Envoy picks the certificate by SNI:
curl -s -o /dev/null -w "%{http_code} verify=%{ssl_verify_result}\n" \
--resolve test.franzramadhan.lab:443:192.168.101.152 https://test.franzramadhan.lab/
# 404 verify=0
Pi-hole for the LAN and the tailnet
Pi-hole runs in the cluster with its DNS on a LoadBalancer IP
(192.168.101.161, TCP and UDP on one Service). The Service uses
externalTrafficPolicy: Local, so the query log shows real client IPs. One
dnsmasq line makes every internal name point at the gateway:
address=/franzramadhan.lab/192.168.101.152
That covers the LAN. For the tailnet, I found that Tailscale devices used
1.1.1.1 as their global resolver and skipped Pi-hole. A second Service with
loadBalancerClass: tailscale gives Pi-hole a 100.x address. I set that
address as the global nameserver of the tailnet (Override DNS servers on). I
also set it as the split DNS target for franzramadhan.lab. Now ad blocking and
.lab names work on my laptop at home or away:
dig doubleclick.net +short
# 0.0.0.0
Issue 1: external-dns Ignored Every Annotation
Symptom
I annotated an HTTPRoute to become a proxied Cloudflare CNAME. external-dns produced a plain, unproxied A record to a private IP instead.
Cause
external-dns 0.2x moved its annotations from external-dns.alpha.kubernetes.io/
to external-dns.kubernetes.io/. external-dns ignores the old prefix and logs
nothing. Also, for Gateway API sources, external-dns reads only target from
the Gateway. cloudflare-proxied must be on the route.
Fix
I used the new prefix, put target on the Gateway and put
cloudflare-proxied on the route. Part 9 uses this to publish apps through a
Cloudflare Tunnel.
Lessons
- Test ACME on staging first. It costs one extra minute and protects the rate limit.
- A private CA is only as useful as its distribution. Put the root into cloud-init, or new nodes will not trust anything.
- “Silently ignored” is the worst failure mode. When a controller does something unexpected, check the annotation names against the version you run.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA (this post)
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
