Back to Home
post-thumb

Homelab Rebuild Part 6: Certificates, DNS and a Private CA

| October 09, 2026 | 6 min to read

Intro

I wanted every service in the rebuilt cluster to have a name and a certificate that my devices trust. That applies to the public domain I own (franzramadhan.dev). It also applies to internal names that will never be public (franzramadhan.lab, which is not a registered domain).

TL;DR: cert-manager issues a Let’s Encrypt wildcard for *.franzramadhan.dev through Cloudflare DNS-01 and runs a private root CA for *.franzramadhan.lab, trusted on every node. Pi-hole resolves the .lab zone for the LAN and, through a Tailscale LoadBalancer, for every tailnet device. external-dns writes Cloudflare records, but only with the new external-dns.kubernetes.io/ annotation prefix.

The Figure

A card being stamped: a certificateLeft: Let's Encrypt die. Right: lab-root-ca die. Cabinet: Pi-hole. Flag: external-dnscert-manager runs two issuers. The ACME issuer gets a wildcard for *.franzramadhan.dev from Let's Encrypt through a Cloudflare DNS-01 TXT record, staging first so a mistake costs no production rate limit. The CA issuer signs *.franzramadhan.lab with lab-root-ca, a self-signed ECDSA root that every node, new VMs through cloud-init and the Mac trust. Pi-hole answers the .lab zone with the gateway, 192.168.101.152, on the LAN and on the tailnet, and external-dns only reads the external-dns.kubernetes.io/ prefix: the old alpha one is silently ignored.

The two presses are the two issuers of cert-manager. The left one carries the public Let’s Encrypt die, fed by the Cloudflare token box, with a selector that stamps on staging before production. The right one carries the house die, wired to the lab-root-ca block on its stand. The cabinet is Pi-hole, with one lamp for a LAN query and one for a tailnet query. The flag on the out box is external-dns, which writes to Cloudflare. Press 1 or 2 (or click a press) to stamp a card. Press S to switch staging and production, 3 for a DNS lookup and 4 to flip the annotation prefix. I built the figure with the anatomy agent skill.

The Situation

  • cert-manager v1.21.2 with Gateway API support
  • Pi-hole 2026.07.2 (chart 2.38.0)
  • external-dns 0.23.0 (chart 1.23.0)
  • lab-gateway (Cilium) on 192.168.101.152 with HTTPS listeners
  • The Cloudflare API token lives in OpenBao (Part 5)

Setup

Public: a Let’s Encrypt wildcard

DNS-01 (proof of domain control through a DNS TXT record) is the only way to get a wildcard. It also works for a cluster that the internet cannot reach. The token needs only Zone DNS Edit and Zone Read on the one zone. I issued from the staging endpoint first, so a mistake does not burn production rate limits:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-franzramadhan-dev
  namespace: gateway
spec:
  secretName: wildcard-franzramadhan-dev
  dnsNames:
    - "*.franzramadhan.dev"
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt

Staging issued in about 90 seconds. Production issued right after.

Internal: a private root CA

cert-manager can bootstrap a CA. A self-signed issuer signs one root certificate, and a CA ClusterIssuer uses it for everything else.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: lab-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: franzramadhan.lab Root CA
  secretName: lab-root-ca
  duration: 87600h # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned

The root certificate goes into the trust store of each node (/usr/local/share/ca-certificates/ plus update-ca-certificates). It also goes into the cloud-init snippet for future VMs and into the keychain on my Mac. The OpenBao listener and the *.franzramadhan.lab gateway listener both use certificates from this CA.

Both listeners share port 443 on the same Gateway, and Envoy picks the certificate by SNI:

curl -s -o /dev/null -w "%{http_code} verify=%{ssl_verify_result}\n" \
  --resolve test.franzramadhan.lab:443:192.168.101.152 https://test.franzramadhan.lab/
# 404 verify=0

Pi-hole for the LAN and the tailnet

Pi-hole runs in the cluster with its DNS on a LoadBalancer IP (192.168.101.161, TCP and UDP on one Service). The Service uses externalTrafficPolicy: Local, so the query log shows real client IPs. One dnsmasq line makes every internal name point at the gateway:

address=/franzramadhan.lab/192.168.101.152

That covers the LAN. For the tailnet, I found that Tailscale devices used 1.1.1.1 as their global resolver and skipped Pi-hole. A second Service with loadBalancerClass: tailscale gives Pi-hole a 100.x address. I set that address as the global nameserver of the tailnet (Override DNS servers on). I also set it as the split DNS target for franzramadhan.lab. Now ad blocking and .lab names work on my laptop at home or away:

dig doubleclick.net +short
# 0.0.0.0

Issue 1: external-dns Ignored Every Annotation

Symptom

I annotated an HTTPRoute to become a proxied Cloudflare CNAME. external-dns produced a plain, unproxied A record to a private IP instead.

Cause

external-dns 0.2x moved its annotations from external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/. external-dns ignores the old prefix and logs nothing. Also, for Gateway API sources, external-dns reads only target from the Gateway. cloudflare-proxied must be on the route.

Fix

I used the new prefix, put target on the Gateway and put cloudflare-proxied on the route. Part 9 uses this to publish apps through a Cloudflare Tunnel.

Lessons

  1. Test ACME on staging first. It costs one extra minute and protects the rate limit.
  2. A private CA is only as useful as its distribution. Put the root into cloud-init, or new nodes will not trust anything.
  3. “Silently ignored” is the worst failure mode. When a controller does something unexpected, check the annotation names against the version you run.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA (this post)
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths
  28. Part 28: Observability, Take Three: Making It Worth Looking At
  29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
  30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
  31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
  32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
  33. Part 33: The Firewall Held, the Tailnet Did Not
  34. Part 34: From Watching to Blocking, and a Record of Who Did What
  35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
  36. Part 36: The Fans Were Loud, the CPUs Were Idle
  37. Part 37: A Power Trip, a Search Domain and a Wildcard
  38. Part 38: An Autonomous AI Pentester in a Sealed Box
  39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References