Homelab Rebuild Part 6: Certificates, DNS and a Private CA

| October 09, 2026 | 6 min to read

Intro

Every service in the rebuilt cluster should have a name and a certificate that my devices trust, both for the public domain I own (franzramadhan.dev) and for internal names that will never be public (franzramadhan.lab, which isn’t a registered domain).

TL;DR: cert-manager issues a Let’s Encrypt wildcard for *.franzramadhan.dev through Cloudflare DNS-01 and runs a private root CA for *.franzramadhan.lab, trusted on every node. Pi-hole resolves the .lab zone for the LAN and, through a Tailscale LoadBalancer, for every tailnet device. external-dns writes Cloudflare records, but only with the new external-dns.kubernetes.io/ annotation prefix.

The Figure

A card being stamped: a certificateLeft: Let's Encrypt die. Right: lab-root-ca die. Cabinet: Pi-hole. Flag: external-dnscert-manager runs two issuers. The ACME issuer gets a wildcard for *.franzramadhan.dev from Let's Encrypt through a Cloudflare DNS-01 TXT record, staging first so a mistake costs no production rate limit. The CA issuer signs *.franzramadhan.lab with lab-root-ca, a self-signed ECDSA root that every node, new VMs through cloud-init and the Mac trust. Pi-hole answers the .lab zone with the gateway, 192.168.101.152, on the LAN and on the tailnet, and external-dns only reads the external-dns.kubernetes.io/ prefix: the old alpha one is silently ignored.

The two presses are cert-manager’s two issuers. The left one carries the public Let’s Encrypt die, fed by the Cloudflare token box, with a selector that stamps on staging before production. The right one carries the house die, wired to the lab-root-ca block on its stand. The cabinet is Pi-hole, with one lamp for a LAN query and one for a tailnet query, and the flag on the out box is external-dns writing to Cloudflare. Press 1 or 2 (or click a press) to stamp a card, S to switch staging and production, 3 for a DNS lookup and 4 to flip the annotation prefix. I built the figure with the anatomy agent skill.

The Situation

cert-manager v1.21.2 with Gateway API support
Pi-hole 2026.07.2 (chart 2.38.0)
external-dns 0.23.0 (chart 1.23.0)
lab-gateway (Cilium) on 192.168.101.152 with HTTPS listeners
The Cloudflare API token lives in OpenBao (Part 5)

Setup

Public: a Let’s Encrypt wildcard

DNS-01 is the only way to get a wildcard, and it works for a cluster that isn’t reachable from the internet. The token needs only Zone DNS Edit and Zone Read on the one zone. I issued from the staging endpoint first, so a mistake doesn’t burn production rate limits:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-franzramadhan-dev
  namespace: gateway
spec:
  secretName: wildcard-franzramadhan-dev
  dnsNames:
    - "*.franzramadhan.dev"
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt

Staging issued in about 90 seconds; production right after.

Internal: a private root CA

cert-manager can bootstrap a CA: a self-signed issuer signs one root certificate, and a CA ClusterIssuer uses it for everything else.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: franzramadhan.lab Root CA
  secretName: lab-root-ca
  duration: 87600h # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned

The root certificate goes into each node’s trust store (/usr/local/share/ca-certificates/ plus update-ca-certificates), into the cloud-init snippet for future VMs, and into my Mac’s keychain. OpenBao’s listener and the *.franzramadhan.lab gateway listener both use certificates from this CA.

Both listeners share port 443 on the same Gateway, and Envoy picks the certificate by SNI:

curl -s -o /dev/null -w "%{http_code} verify=%{ssl_verify_result}\n" \
  --resolve test.franzramadhan.lab:443:192.168.101.152 https://test.franzramadhan.lab/
# 404 verify=0

Pi-hole for the LAN and the tailnet

Pi-hole runs in the cluster with its DNS on a LoadBalancer IP (192.168.101.161, TCP and UDP on one Service) and externalTrafficPolicy: Local, so its query log shows real client IPs. One dnsmasq line makes every internal name point at the gateway:

address=/franzramadhan.lab/192.168.101.152

That covers the LAN. For the tailnet, I found that Tailscale devices used 1.1.1.1 as their global resolver and skipped Pi-hole entirely. A second Service with loadBalancerClass: tailscale gives Pi-hole a 100.x address, set as the tailnet’s global nameserver (Override DNS servers on) and as the split DNS target for franzramadhan.lab. Now ad blocking and .lab names work on my laptop at home or away:

dig doubleclick.net +short
# 0.0.0.0

Issue 1: external-dns Ignored Every Annotation

Symptom

An HTTPRoute annotated to become a proxied Cloudflare CNAME produced a plain, unproxied A record to a private IP.

Cause

external-dns 0.2x moved its annotations from external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/. The old prefix is silently ignored. On top of that, for Gateway API sources only target is read from the Gateway, and cloudflare-proxied must be on the route.

Fix

New prefix, target on the Gateway, cloudflare-proxied on the route. Part 9 uses this to publish apps through a Cloudflare Tunnel.

Lessons

Test ACME on staging first. It costs one extra minute and protects the rate limit.
A private CA is only as useful as its distribution. Put the root into cloud-init, or new nodes won’t trust anything.
“Silently ignored” is the worst failure mode. When a controller does something unexpected, check the annotation names against the version you run.

Series

This post is part of Homelab Rebuild: