Homelab Rebuild Part 31: First Backups, Postgres, OpenBao and etcd to R2

Intro

Every part of this series so far made the lab more available: two Postgres instances on two hosts, three OpenBao pods in a Raft cluster, three controllers. None of that is a backup. A replica copies a bad DELETE in milliseconds, and bao kv metadata delete on a Raft cluster deletes the secret on all three pods at once.

So this post is about the boring part: getting copies of the data out of the house, and proving they come back.

TL;DR: Three backups now go to one Cloudflare R2 bucket. Postgres archives every WAL segment and takes a daily base backup through the CNPG Barman Cloud plugin, for 14 days of point-in-time recovery. OpenBao takes a daily Raft snapshot from a CronJob. k0s (etcd, PKI and the Secret encryption key) is backed up daily from a controller, encrypted with age. Kyverno blocked the first Postgres backup, Tetragon flagged all of them, and the first restore failed three times before it worked. All three restores passed in the end.

The Figure

The figure is a tape archive, because that is how point-in-time recovery works. The deck at the left records the database’s write-ahead log onto tape without stopping, one segment at a time. Once a day a full reel (the base backup) goes into the rack, and the rack keeps 14 days of them. The two sealed canisters on the top shelf are the OpenBao snapshot and the age-encrypted k0s backup: they can only be opened with a key that is not in the room. Turn the counter (drag the knob, or use the arrow keys) to pick a moment, and the readout shows which reel and how much tape a restore to that moment needs. Press R to run the restore. I built it with the anatomy skill (see References).

The Situation

Nothing was backed up. “Add barman-cloud when there is an S3 bucket” had been on the to-do list since the Postgres cluster went in, and the bucket never came.

What actually holds state in the lab:

Everything else is in git, and Flux puts it back.

Why not Velero

Velero is the usual answer, and I decided against it. The manifests are already in git and the secrets are in OpenBao, so a Velero backup of the objects would be a third copy of things I already have. What is left is data inside running databases, and a file-level copy of a Postgres volume while Postgres writes to it is neither consistent nor point-in-time. Each store has its own backup tool that does this properly, so each store gets that tool.

flowchart LR
  PG["CNPG pg"] -->|"WAL, continuous"| R2["R2 bucket"]
  PG -->|"base backup, 12:00"| R2
  BAO["OpenBao Raft"] -->|"snapshot, 12:30"| R2
  K0S["k0s controller-01"] -->|"age-encrypted, 13:00"| R2
  R2 -.->|"restore test"| T["throwaway copies"]

The cluster is off from 23:30 to 06:00 every night (Part 10), so every schedule sits in the middle of the day.

Setup

The bucket

One R2 bucket, homelab-backups, with prefixes pg/, openbao/ and k0s/. R2 has no egress fees, which matters for restores, and the free tier covers 10 GB. The token is scoped to that one bucket with Object Read & Write:

GET /homelab-backups?list-type=2   200
GET /?x-id=ListBuckets             403

It can’t even list the other buckets in the account. The token sits in OpenBao twice, at secret/databases/r2-backup and secret/openbao/r2-backup, because a Kyverno policy from Part 11 only lets an ExternalSecret read the path of its own namespace.

Postgres: the Barman Cloud plugin

CNPG 1.30 still has the in-tree barmanObjectStore, but it is deprecated in favour of the Barman Cloud plugin , which runs as a sidecar in each instance pod. I installed chart 0.8.1 (plugin v0.15.1) in cnpg-system after checking its digest against the repository index. The store:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: r2
spec:
  retentionPolicy: "14d"
  configuration:
    destinationPath: s3://homelab-backups/pg/
    endpointURL: https://<account id>.r2.cloudflarestorage.com
    s3Credentials:
      accessKeyId: {name: r2-backup, key: access_key_id}
      secretAccessKey: {name: r2-backup, key: secret_access_key}
    wal:
      compression: gzip
    data:
      compression: gzip
  instanceSidecarConfiguration:
    env:
      # boto3 >= 1.36 sends checksums some S3-compatible stores reject
      - {name: AWS_REQUEST_CHECKSUM_CALCULATION, value: when_required}
      - {name: AWS_RESPONSE_CHECKSUM_VALIDATION, value: when_required}

The cluster points at it, which makes the plugin the WAL archiver:

spec:
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: r2

And a daily base backup. CNPG’s cron has a seconds field and runs in UTC, so 12:00 WIB is 0 0 5 * * *:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: pg-daily
spec:
  schedule: "0 0 5 * * *"
  immediate: true
  cluster:
    name: pg
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Adding the plugin restarts both instances once (the sidecar is new), with one switchover. LiteLLM logged “key lookup did not answer within 10s” during it, and its request-error alert fired.

OpenBao: a Raft snapshot CronJob

OpenBao has no backup operator, but bao operator raft snapshot save writes a consistent snapshot of the whole store. The CronJob runs it at 12:30 WIB in two containers:

1. snapshot (the OpenBao image) logs in with a Kubernetes auth role, raft-snapshot, whose policy allows exactly one thing:

   path "sys/storage/raft/snapshot" {
     capabilities = ["read"]
   }
   

   It writes the snapshot to an emptyDir. Only ca.crt is projected from the OpenBao TLS secret, not the private key.

2. upload (rclone) copies it to r2:homelab-backups/openbao/ and pings a healthchecks.io check (/fail on error).

The snapshot is encrypted by OpenBao’s barrier, so the copy in R2 is useless without the unseal key, which is not in R2. A bucket lifecycle rule deletes snapshots after 30 days.

etcd: an encrypted k0s backup

The k0s controllers are not Kubernetes nodes, so this one is a systemd timer on k0s-controller-01, not a CronJob. k0s backup writes the etcd snapshot, the PKI and k0s.yaml. I add /etc/k0s/encryption-config.yaml (the Secret encryption key from Part 26), because the etcd snapshot is useless without it.

That bundle holds the cluster CA’s private key, so it never leaves the host unencrypted. It is encrypted with age to a public key whose private half exists only on my laptop:

k0s backup --data-dir /opt/lib/k0s --save-path "$work"
tar -cf - -C "$work" "$tarball" -C /etc/k0s encryption-config.yaml \
  | age -r "$AGE_RECIPIENT" > "$out"
curl -fsS --aws-sigv4 "aws:amz:auto:s3" \
  --user "$R2_ACCESS_KEY_ID:$R2_SECRET_ACCESS_KEY" \
  -T "$out" "$R2_ENDPOINT/$R2_BUCKET/k0s/$(basename "$out")"

curl --aws-sigv4 signs the upload, so the host needs no S3 client. The timer runs at 13:00 WIB with Persistent=true, so a run missed while the VM was off happens after the next boot.

Issue 1: Kyverno blocked the first base backup

Symptom

The WAL archiver worked at once (about 4 seconds per segment). The immediate base backup failed:

error: unable to upgrade connection: admission webhook
"vpol.validate.kyverno.svc-ignore-43ea372b" denied the request:
Policy protect-sensitive-exec failed: exec/attach/port-forward into databases
is limited to cluster admins

Cause

My own policy from Part 11. Only system:masters may exec into the openbao, databases and cert-manager namespaces, so a compromised ServiceAccount with pods/exec can’t get a shell next to the data. The CNPG operator starts a plugin base backup by running /controller/manager backup <name> in the instance pod, and that is a pods/exec.

Fix

Allow the operator’s ServiceAccount, and only in databases:

validations:
  - expression: >-
      "system:masters" in request.userInfo.groups ||
      (request.namespace == "databases" &&
       request.userInfo.username == "system:serviceaccount:cnpg-system:cnpg-cloudnative-pg")

A manual backup right after took 16 seconds.

Issue 2: Tetragon flagged every backup, and the first fix did nothing

Symptom

The “Tetragon policy hit” alert fired for unexpected-egress (Part 11’s exfiltration detector) on every barman call from databases, and on the snapshot job’s healthchecks ping from openbao:

policy=unexpected-egress ns=databases bin=/venv/bin/barman-cloud-wal-archive n=5
policy=unexpected-egress ns=databases bin=/venv/bin/barman-cloud-wal-restore n=38
policy=unexpected-egress ns=openbao   bin=/usr/bin/wget n=1

The policy flags any TCP connect from those namespaces to an address outside the cluster and LAN ranges, and R2 is on Cloudflare (172.64.x.x).

First fix

Give the Cloudflare ranges to the barman binaries only, with a second selector, so any other process in a database pod is still flagged:

selectors:
- matchBinaries:
  - operator: NotPrefix
    values: [/venv/bin/barman-cloud-]
  matchArgs: [{index: 0, operator: NotDAddr, values: [<cluster and LAN>]}]
- matchBinaries:
  - operator: Prefix
    values: [/venv/bin/barman-cloud-]
  matchArgs: [{index: 0, operator: NotDAddr, values: [<cluster, LAN, Cloudflare>]}]

Tetragon reloaded it at 14:03:00 UTC. The next WAL archive, at 14:03:46, fired again, and so did the one at 14:08:47. The destination was 172.64.190.1, inside the allowed 172.64.0.0/13.

Cause

The event said binary: /venv/bin/barman-cloud-wal-archive, but the argument list started with that same path. The barman tools are Python scripts. The kernel runs the interpreter from the shebang, and Tetragon’s in-kernel matchBinaries compares against the file that is actually executing: /venv/bin/python3, which resolves to /usr/bin/python3.13 in the distroless sidecar. The event shows the script path for humans. The match uses the interpreter.

Fix

Add the interpreter as a second prefix in both selectors:

values: [/venv/bin/barman-cloud-, /usr/bin/python3]

Zero hits after the next archive. The OpenBao snapshot job got a podSelector exception instead (label app=raft-snapshot), because the healthchecks.io servers sit on hosting IPs that change, and the snapshot it uploads is barrier-encrypted anyway.

Issue 3: The first restore failed three times

Symptom

A backup only counts once it has been restored, so I created a throwaway cluster from R2:

bootstrap:
  recovery:
    source: origin
externalClusters:
  - name: origin
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters: {barmanObjectName: r2, serverName: pg}

It went healthy after 4 minutes 21 seconds, but only after three recovery pods had died with:

encountered an error while checking the presence of first needed WAL

Cause

Timing, not a broken backup. The base backup had finished at 13:45:51. To be consistent it needs the WAL segment that was being written when it ended, and Postgres archives a segment only when it switches to the next one. On a quiet database that happens at archive_timeout, about 5 minutes. The first recovery pods started before that segment was in R2. The fourth found it.

Result

The restored litellm database matched the live one exactly:

pg-2:              28 spend rows | 1 key | 13 MB
pg-restore-test-1: 28 spend rows | 1 key | 13 MB

Restoring OpenBao and k0s

OpenBao. I started a single-node OpenBao in a scratch pod: its own Raft storage, a listener on localhost only, no retry_join, and the same static seal key. After bao operator init and bao operator raft snapshot restore -force, it unsealed itself with the static key, and bao status showed the same cluster ID as the real one. All 11 KV namespaces were there, and the token I had used that evening (it was in the snapshot) could read a secret written an hour earlier.

k0s. I downloaded the object to the laptop, decrypted it with the age key and listed it: etcd-snapshot.db, k0s.yaml, the k0s manifests, the PKI (29 files plus 8 for etcd) and encryption-config.yaml. I didn’t restore etcd into a scratch controller; k0s restore on a fresh VM is the next test.

Result

Retention: 14 days of Postgres recovery (the plugin deletes older backups), 30 days of snapshots (bucket lifecycle rules). Everything fits the R2 free tier.

Knowing it still works

A backup that silently stopped is worse than none, because you rely on it.

• Postgres: Grafana alerts when the last base backup is older than 26 hours, when a backup fails, and when more than 5 WAL segments wait for the archiver for 15 minutes.
• OpenBao: the CronJob pings a healthchecks.io check on success and /fail on error. The check expects a ping daily at 12:30 with 2 hours of grace, so a run that never starts is caught too.
• Grafana itself: an always-firing Watchdog rule pings another check every two minutes. If the lab, Grafana or VictoriaMetrics dies, the email comes from outside the house.

Lessons

1. Replicas are not backups. Two instances and a three-pod Raft cluster replicate a mistake as fast as they replicate data.
2. Use each store’s own backup tool. WAL archiving and Raft snapshots are consistent. A file copy of a running database volume is not.
3. Your own guardrails will block the backup. Both blocks came from my policies: Kyverno’s exec rule and Tetragon’s egress detector. Make each exception as narrow as the tool allows.
4. Tetragon matches the interpreter, not the script. For Python or shell tools, matchBinaries needs the interpreter’s real path, even though the event shows the script.
5. A restore right after a backup waits for one WAL switch. Retry, or force one with pg_switch_wal().
6. Encrypt what holds keys. The k0s backup carries the cluster CA, so it is encrypted to a key that never touches the cluster.
7. It is not a backup until it has been restored. Each test here found something: the WAL timing, and that the OpenBao snapshot needs the seal key next to it.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2 (this post)

References

• GitHub: wheresryan22/anatomy
• CloudNativePG: Barman Cloud plugin
• CloudNativePG: Recovery
• Cloudflare R2: S3 API compatibility
• OpenBao: Integrated storage snapshots
• k0s: Backup and restore
• Tetragon: Selectors, matchBinaries
• age: a simple, modern file encryption tool
• Velero