Homelab Rebuild Part 6: Certificates, DNS and a Private CA

| October 09, 2026 | 4 min to read

Intro

Every service in the rebuilt cluster should have a name and a certificate that my devices trust, both for the public domain I own (franzramadhan.dev) and for internal names that will never be public (franzramadhan.lab, which isn’t a registered domain).

TL;DR: cert-manager issues a Let’s Encrypt wildcard for *.franzramadhan.dev through Cloudflare DNS-01 and runs a private root CA for *.franzramadhan.lab, trusted on every node. Pi-hole resolves the .lab zone for the LAN and, through a Tailscale LoadBalancer, for every tailnet device. external-dns writes Cloudflare records, but only with the new external-dns.kubernetes.io/ annotation prefix.

The Situation

cert-manager v1.21.2 with Gateway API support
Pi-hole 2026.07.2 (chart 2.38.0)
external-dns 0.23.0 (chart 1.23.0)
lab-gateway (Cilium) on 192.168.101.152 with HTTPS listeners
The Cloudflare API token lives in OpenBao (Part 5)

Setup

Public: a Let’s Encrypt wildcard

DNS-01 is the only way to get a wildcard, and it works for a cluster that isn’t reachable from the internet. The token needs only Zone DNS Edit and Zone Read on the one zone. I issued from the staging endpoint first, so a mistake doesn’t burn production rate limits:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-franzramadhan-dev
  namespace: gateway
spec:
  secretName: wildcard-franzramadhan-dev
  dnsNames:
    - "*.franzramadhan.dev"
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt

Staging issued in about 90 seconds; production right after.

Internal: a private root CA

cert-manager can bootstrap a CA: a self-signed issuer signs one root certificate, and a CA ClusterIssuer uses it for everything else.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: franzramadhan.lab Root CA
  secretName: lab-root-ca
  duration: 87600h # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned

The root certificate goes into each node’s trust store (/usr/local/share/ca-certificates/ plus update-ca-certificates), into the cloud-init snippet for future VMs, and into my Mac’s keychain. OpenBao’s listener and the *.franzramadhan.lab gateway listener both use certificates from this CA.

Both listeners share port 443 on the same Gateway, and Envoy picks the certificate by SNI:

curl -s -o /dev/null -w "%{http_code} verify=%{ssl_verify_result}\n" \
  --resolve test.franzramadhan.lab:443:192.168.101.152 https://test.franzramadhan.lab/
# 404 verify=0

Pi-hole for the LAN and the tailnet

Pi-hole runs in the cluster with its DNS on a LoadBalancer IP (192.168.101.161, TCP and UDP on one Service) and externalTrafficPolicy: Local, so its query log shows real client IPs. One dnsmasq line makes every internal name point at the gateway:

address=/franzramadhan.lab/192.168.101.152

That covers the LAN. For the tailnet, I found that Tailscale devices used 1.1.1.1 as their global resolver and skipped Pi-hole entirely. A second Service with loadBalancerClass: tailscale gives Pi-hole a 100.x address, set as the tailnet’s global nameserver (Override DNS servers on) and as the split DNS target for franzramadhan.lab. Now ad blocking and .lab names work on my laptop at home or away:

dig doubleclick.net +short
# 0.0.0.0

Issue 1: external-dns Ignored Every Annotation

Symptom

An HTTPRoute annotated to become a proxied Cloudflare CNAME produced a plain, unproxied A record to a private IP.

Cause

external-dns 0.2x moved its annotations from external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/. The old prefix is silently ignored. On top of that, for Gateway API sources only target is read from the Gateway, and cloudflare-proxied must be on the route.

Fix

New prefix, target on the Gateway, cloudflare-proxied on the route. Part 9 uses this to publish apps through a Cloudflare Tunnel.

Lessons

Test ACME on staging first. It costs one extra minute and protects the rate limit.
A private CA is only as useful as its distribution. Put the root into cloud-init, or new nodes won’t trust anything.
“Silently ignored” is the worst failure mode. When a controller does something unexpected, check the annotation names against the version you run.

Series

This post is part of Homelab Rebuild: