Loading

Back to Home

Homelab Rebuild Part 5: Secrets Without Secrets in Git

Technology Kubernetes Homelab Secrets
| October 08, 2026 | 4 min to read

Intro

The old repo had a Pi-hole admin password and a Tailscale OAuth client secret committed in plain text. For the rebuild, the rule was simple: no credential in git, ever. That needs somewhere else to keep them.

TL;DR: OpenBao 2.7.1 as a 3-node Raft cluster with static auto-unseal, External Secrets Operator to turn OpenBao entries into Kubernetes Secrets, and the agent injector for workloads that should never have a Secret object. Every secret enters through stdin, never through arguments or the terminal.

The Situation

  • OpenBao chart 0.30.3 (OpenBao 2.7.1)
  • External Secrets Operator 2.12.0
  • local-path-provisioner v0.0.37 as the StorageClass
  • No Longhorn (I had removed it). Raft replicates OpenBao’s data itself, so node-local disks are enough.

Setup

OpenBao on Raft

Three server pods, one per Proxmox host, joined through retry_join:

storage "raft" {
  path = "/openbao/data"
  retry_join { leader_api_addr = "https://openbao-0.openbao-internal:8200" }
  retry_join { leader_api_addr = "https://openbao-1.openbao-internal:8200" }
  retry_join { leader_api_addr = "https://openbao-2.openbao-internal:8200" }
}

The StatefulSet is OrderedReady, so openbao-1 and openbao-2 start only after openbao-0 is initialized and unsealed.

Static auto-unseal, and its trade-off

I didn’t want to type three key shares after every pod restart, and I had no KMS handy. OpenBao’s static seal reads a 32-byte key from a file:

seal "static" {
  current_key_id = "lab-2026-10-12"
  current_key    = "file:///openbao/userconfig/openbao-seal-key/key"
}

The key lives in a Kubernetes Secret mounted into the pods, with a backup copy outside the cluster. The trade-off is explicit: anyone with cluster-admin can read the unseal key. For a single-person homelab that’s acceptable; for anything shared it isn’t.

With auto-unseal, bao operator init returns recovery keys instead of unseal keys. I wrote the output straight to a mode-600 file instead of the terminal:

(umask 077; kubectl -n openbao exec openbao-0 -- \
  bao operator init -recovery-shares=5 -recovery-threshold=3 -format=json > init.json)

Secrets go in through stdin

Every write to OpenBao follows the same pattern: the token and the value travel through stdin, so they never appear in process arguments, shell history or the terminal:

{ jq -r .root_token init.json; printf %s "$VALUE"; } \
  | kubectl -n openbao exec -i openbao-0 -- sh -c \
    'read -r BAO_TOKEN; export BAO_TOKEN; bao kv put -mount=secret my-app/db password=-'

External Secrets Operator

ESO authenticates to OpenBao with Kubernetes auth and a read-only policy, and apps declare what they need:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: pihole-admin
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: openbao
  target:
    name: pihole-admin
  data:
    - secretKey: password
      remoteRef:
        key: pihole/admin
        property: password

Today this feeds the Pi-hole password, the Cloudflare token for cert-manager and external-dns, and the Cloudflare Tunnel token.

Issue 1: The Injector Ignored My Pod

Symptom

A test pod with openbao.hashicorp.com/agent-inject: "true" started without any sidecar.

Cause

The chart’s injector image is still hashicorp/vault-k8s, which watches the vault.hashicorp.com/ annotation prefix. The injected agent itself is OpenBao (quay.io/openbao/openbao).

Fix

Use the vault.hashicorp.com/ annotations. The secret landed in /vault/secrets/greeting as expected.

Issue 2: Helm Wouldn’t Upgrade the Chart

Symptom

UPGRADE FAILED: conflict occurred while applying object /openbao-agent-injector-cfg ...
conflict with "vault-k8s": .webhooks[name="vault.hashicorp.com"].clientConfig.caBundle

Cause

Helm 4 applies server-side. The injector writes its own webhook caBundle at runtime, so it owns that field.

Fix

helm upgrade --force-conflicts. The injector rewrites the caBundle right away.

Lessons

  1. Decide the unseal trade-off on purpose. Static seal buys unattended restarts by putting the key next to the data.
  2. stdin is the secret channel. Arguments end up in ps, history and logs.
  3. Forks keep their parents’ names in odd places. OpenBao’s injector still answers to vault.hashicorp.com/.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git (this post)
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers

References