Homelab Rebuild Part 26: TLS on the Front, Plaintext Behind It

Intro

cert-manager has been issuing certificates since Part 6 : Let’s Encrypt for *.franzramadhan.dev, and a private lab CA for *.franzramadhan.lab. So I assumed the homelab was “on TLS”. This morning I checked every endpoint to be sure, including the internal ones.

The front doors were fine. What was behind them wasn’t: once a request got past the gateway, it often went on in plain HTTP, sometimes across the router from Part 25 . Postgres accepted logins without TLS. And every Kubernetes Secret was stored in etcd as plain text.

TL;DR: Every URL had a valid certificate, but TLS ended at the gateway. I turned on Cilium WireGuard for pod traffic between hosts, made lab-gateway’s port 80 a 301 to HTTPS, made Postgres reject non-TLS logins with a lab-CA server certificate, and encrypted Secrets in etcd with secretbox. Two traps cost me downtime: Cilium only picks up the change after an agent restart (about 1.5 minutes without a gateway each time), and k0s runs kube-apiserver as its own user, so a root-only key file kept two controllers down.

The Figure

The figure is an office tube post. The front counter is lab-gateway, with hatch 80 and hatch 443 and the lab-CA certificate plate next to it. The three stations at the back are the Proxmox hosts, and capsules travel between them the way pod traffic crosses hosts. The box on the right is Postgres, and the drawer cabinet is etcd. It starts in the old state: capsules travel open, the Postgres box takes any capsule, and the drawer holds plain sheets. W seals the capsules (WireGuard), H sends a slip from hatch 80 round to 443 (the 301), P makes the Postgres box drop open capsules into its reject tray, and E files the Secrets in sealed envelopes, with the key’s two tags on the cabinet. A flips everything at once. I generated it with the anatomy agent skill.

The Situation

• k0s v1.36.4: 3 controllers (etcd members) and 5 workers on 3 Proxmox hosts.
• Cilium 1.20.2: native routing, kube-proxy replacement, Gateway API. lab-gateway on 192.168.101.152 for the LAN, plus Tailscale Ingresses for the tailnet.
• cert-manager with a lab-ca ClusterIssuer: root CA valid until 2036, 90-day wildcard certificate for *.franzramadhan.lab.
• CloudNativePG 1.30.1, cluster pg (PostgreSQL 18.6), primary plus one replica on two hosts.
• OpenBao for secrets, with External Secrets syncing them into Kubernetes Secrets.

The audit

I listed what can be reached and how:

kubectl get clusterissuers,certificates -A
kubectl get gateway -A -o json | jq ...            # listeners and their certs
kubectl get httproute,ingress -A
kubectl get svc -A --field-selector spec.type=LoadBalancer

Then I checked each URL against the lab CA itself, not by trusting the browser:

curl --cacert lab-ca.crt --resolve grafana.franzramadhan.lab:443:192.168.101.152 \
  -o /dev/null -w '%{http_code} verify=%{ssl_verify_result}\n' https://grafana.franzramadhan.lab/
# 302 verify=0

The Postgres row surprised me. CloudNativePG turns SSL on and uses certificate login for replication, but its last pg_hba rule is host all all all scram-sha-256. host matches TLS and non-TLS alike.

flowchart LR
  B[Browser] -->|HTTPS| GW[lab-gateway<br/>Envoy on a worker]
  GW -->|HTTP| P1[Grafana pod<br/>other host]
  P1 -->|HTTP| VM[VictoriaMetrics<br/>other host]
  APP[App pod] -->|TLS optional| PG[(Postgres pg)]
  API[kube-apiserver] -->|plain| ETCD[(etcd)]

Issue 1: WireGuard on, but nothing encrypted

Symptom

The change was three lines of Cilium values:

encryption:
  enabled: true
  type: wireguard
  nodeEncryption: false   # pod traffic only; Tailscale also runs on the nodes

Flux applied it and the HelmRelease was Ready. cilium-config had enable-wireguard: "true". Every agent still said:

cilium-dbg status | grep Encryption
# Encryption:              Disabled

Cause

The agents read cilium-config at start. Changing the ConfigMap doesn’t restart them unless the chart’s rollOutCiliumPods is on, and it was off.

Fix

A rolling restart, then rollOutCiliumPods: true and operator.rollOutPods: true in values so the next change restarts by itself:

kubectl -n kube-system rollout restart ds/cilium deploy/cilium-operator
cilium-dbg status | grep Encryption
# Encryption:  Wireguard  [NodeEncryption: Disabled, cilium_wg0 (Pubkey: ..., Port: 51871, Peers: 4)]

All five workers show four peers. The controllers aren’t Cilium nodes in k0s. Each peer had a recent handshake and traffic in both directions, between 245 and 436 KB per peer within the first minutes.

Side effect: 1.5 minutes without a gateway

Right after the restart, every HTTPS URL failed:

connect to 192.168.101.152 port 443 failed: Connection refused

The Service still pointed at Envoy’s listener, but Envoy had no gateway listeners until the new agent pushed them again. About 90 seconds later:

lds: add/update listener 'gateway/cilium-gateway-lab-gateway/listener'

It happened again on the second restart. On this cluster a cilium-agent restart means about 1.5 minutes without lab-gateway, so I now plan Cilium changes for quiet hours.

Issue 2: A ConfigMap key no upgrade would remove

Symptom

I also turned off Cilium’s Ingress controller, since everything uses the Gateway API. Its LoadBalancer on .153 went away, but cilium-operator kept logging:

failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden:
User "system:serviceaccount:kube-system:cilium-operator" cannot list resource "ingresses"

The chart had removed the operator’s Ingress permissions, but the live cilium-config still said enable-ingress-controller: "true".

Cause

The chart no longer renders that key when the feature is off. The key belongs to the field manager of the Helm CLI, which installed Cilium before Flux took over (Part 16 ). Server-side apply only removes fields its own manager owns, so the key stayed. I hit the same class of problem with Pi-hole in Part 24 , where one upgrade with serverSideApply: disabled fixed it. Here that didn’t help either. A client-side three-way merge deletes fields that were in the previous release’s manifest, and the key had already gone from that manifest in the first upgrade.

Fix

Remove the key from the live object and restart only the operator, which doesn’t touch the gateway:

kubectl -n kube-system patch cm cilium-config --type=json \
  -p '[{"op":"remove","path":"/data/enable-ingress-controller"}]'
kubectl -n kube-system rollout restart deploy/cilium-operator

HTTP to HTTPS

Before, port 80 on lab-gateway answered 404 for everything. Now one HTTPRoute on the http listener redirects:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
spec:
  parentRefs:
    - name: lab-gateway
      sectionName: http
  hostnames:
    - "*.franzramadhan.lab"
    - "*.franzramadhan.dev"
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

curl -s -o /dev/null -w '%{http_code} -> %{redirect_url}\n' http://grafana.franzramadhan.lab/
# 301 -> https://grafana.franzramadhan.lab:443/

Postgres: TLS or nothing

No app uses the cluster yet, so this was the cheapest moment to make it strict. Two changes: a server certificate from the lab CA, so clients can use sslmode=verify-full, and pg_hba rules that turn non-TLS away.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pg-server
spec:
  secretName: pg-server
  secretTemplate:
    labels:
      cnpg.io/reload: "true"   # PostgreSQL reloads when cert-manager renews
  usages: [server auth]
  dnsNames: [pg-rw, pg-rw.databases.svc, pg-ro, pg-ro.databases.svc, ...]
  duration: 2160h
  issuerRef: {kind: ClusterIssuer, name: lab-ca}

# Cluster pg
certificates:
  serverTLSSecret: pg-server
  serverCASecret: pg-server
postgresql:
  pg_hba:
    - hostnossl all all all reject
    - hostssl all all all scram-sha-256

CloudNativePG put both rules ahead of its default host rule and applied them with a reload: no restart and no failover. Replication keeps CloudNativePG’s own client CA and certificate login. From the replica pod to pg-rw:

psql "host=pg-rw.databases.svc dbname=homelab user=homelab sslmode=disable"
# FATAL:  pg_hba.conf rejects connection for host "10.0.0.160", user "homelab", database "homelab", no encryption
psql "host=pg-rw.databases.svc dbname=homelab user=homelab sslmode=verify-full sslrootcert=server-ca.crt" \
  -c "select ssl, version from pg_stat_ssl where pid = pg_backend_pid()"
# t | TLSv1.3

Issue 3: The key file kube-apiserver couldn’t read

Encrypting Secrets at rest is a kube-apiserver flag, --encryption-provider-config, pointing at a file that holds the key. In k0s, flags go into spec.api.extraArgs. kube-apiserver is a host process on each controller, so the file has to be on all three.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - secretbox:
          keys:
            - name: key-2026-10-17
              secret: <32 random bytes, base64>
      - identity: {}   # existing plain Secrets stay readable

Before turning it on, the key went to two places: OpenBao, and a break-glass file on my laptop. OpenBao runs inside this cluster. If the cluster is broken badly enough to need the key, OpenBao may be down too.

Symptom

I rolled the controllers one at a time. On controller-01, k0s restarted and the API never came back:

error reading encryption provider configuration file "/etc/k0s/encryption-config.yaml":
open /etc/k0s/encryption-config.yaml: permission denied

Cause

k0s runs kube-apiserver as its own user:

id kube-apiserver
# uid=994(kube-apiserver) gid=986(kube-apiserver)

I had written the file as root, mode 0600, which was right for a key and unreadable for the process that needs it. My next step repeated the mistake on controller-02, because the sed command meant to fix my script failed and I didn’t stop to check. Each time the other two controllers kept etcd quorum and the API, so the cluster stayed up.

Fix

chown kube-apiserver:root /etc/k0s/encryption-config.yaml
chmod 400 /etc/k0s/encryption-config.yaml
# readyz 401 (anonymous off, so: up) after 9 seconds

After each controller, the gate: [+]etcd ok, three etcd members, and k0s’s controller count back at 3. Then all existing Secrets get rewritten, so they’re stored again under the new provider:

kubectl get secrets -A -o json | kubectl replace -f -
# 120 replaced

k0s doesn’t ship etcdctl, so I asked etcd directly over its v3 HTTP API with the apiserver’s client certificate:

curl -s --cacert pki/etcd/ca.crt --cert pki/apiserver-etcd-client.crt --key pki/apiserver-etcd-client.key \
  https://127.0.0.1:2379/v3/kv/range -d "{\"key\":\"$(printf /registry/secrets/databases/pg-server | base64)\"}"
# value starts with: k8s:enc:secretbox:v1:key-2026-

Result

flowchart LR
  B[Browser] -->|HTTPS| GW[lab-gateway]
  GW -->|WireGuard between hosts| P1[Grafana pod]
  P1 -->|WireGuard| VM[VictoriaMetrics]
  APP[App pod] -->|TLS 1.3 required| PG[(Postgres pg)]
  API[kube-apiserver] -->|secretbox| ETCD[(etcd)]

Not done:

• Disk encryption. LUKS on the VM disks or the hosts needs a key at every boot, and the cluster starts on its own at 06:00 every morning. A key on the same disk protects nothing. I’ll encrypt backups instead when they go to S3.
• Proxmox UIs still use Proxmox’s own CA. Lab-CA certificates for them are a separate host-level job.

Lessons

1. A certificate on the front door says nothing about the hallway. Check where TLS ends, not only that it starts.
2. Read the running state, not the config. cilium-config said WireGuard was on. The agents said it wasn’t.
3. Know what a restart costs. Here a Cilium agent restart means about 1.5 minutes without a gateway, so Cilium changes now go to quiet hours.
4. Fields from an old Helm CLI owner can outlive every upgrade mode. Once the chart stops rendering a key, neither apply mode removes it.
5. Check which user reads a secret file. Root-only is the safe default and still wrong when the reader is kube-apiserver.
6. Store an encryption key twice before turning encryption on, and once outside the system it protects.
7. Fix a script before running it again. One unchecked sed turned one controller outage into two.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths

References

• GitHub: wheresryan22/anatomy
• Cilium: WireGuard Transparent Encryption
• Kubernetes Gateway API: HTTP redirects and rewrites
• CloudNativePG: Certificates
• CloudNativePG: PostgreSQL configuration, pg_hba
• PostgreSQL: The pg_hba.conf file
• Kubernetes: Encrypting confidential data at rest
• k0s: Configuration options