Homelab Rebuild Part 31: First Backups, Postgres, OpenBao and etcd to R2

| October 18, 2026 | 14 min to read

Intro

Every part of this series so far made the lab more available. I have two Postgres instances on two hosts, three OpenBao pods in a Raft cluster, and three controllers. None of that is a backup. A replica copies a bad DELETE in milliseconds, and bao kv metadata delete on a Raft cluster deletes the secret on all three pods at once.

So this post is about the boring part. I get copies of the data out of the house, and I prove that they come back.

TL;DR: Three backups now go to one Cloudflare R2 bucket. Postgres archives every WAL segment and takes a daily base backup through the CNPG Barman Cloud plugin, for 14 days of point-in-time recovery. OpenBao takes a daily Raft snapshot from a CronJob. A controller backs up k0s (etcd, PKI and the Secret encryption key) daily and encrypts it with age. Kyverno blocked the first Postgres backup, and Tetragon flagged all of them. The first restore failed three times before it worked. All three restores passed in the end.

The Figure

The base backup a restore starts from (its can and its slot dot)Sealed canisters: OpenBao snapshot, encrypted k0s backupDrag the deck's knob sideways, or use the arrow keys (shift for a day), to choose how far back to restore; the counter shows days and hours. The lit can is the newest base backup at or before that moment, and the readout shows how much WAL is replayed on top of it. R runs the restore. The deck never stops recording: every WAL segment goes to R2 as soon as Postgres closes it, at least every 5 minutes (12 segments an hour), and a base backup is taken daily at 12:00 WIB. The rack shows the steady state with 14 days kept; the plugin deletes older backups. The canisters have no counter: OpenBao and k0s are restored from their daily snapshot, not to a point in time.

The figure is a tape archive, because that is how point-in-time recovery works. The deck at the left records the database’s write-ahead log (WAL, the log of every change) onto tape without stopping, one segment at a time. Once a day a full reel (the base backup) goes into the rack, and the rack keeps 14 days of them. The two sealed canisters on the top shelf are the OpenBao snapshot and the age-encrypted k0s backup. Only a key that is not in the room opens them. Turn the counter (drag the knob, or use the arrow keys) to pick a moment. The readout then shows which reel and how much tape a restore to that moment needs. Press R to run the restore. I built it with the anatomy skill (see References).

The Situation

Nothing was backed up. “Add barman-cloud when there is an S3 bucket” was on the to-do list since the Postgres cluster went in, and the bucket never came.

What holds state in the lab:

Store	What is in it	Lost means
CloudNativePG `pg`	`litellm` (keys, spend logs), `homelab` (queues, key/value)	LiteLLM keys and history gone
OpenBao (Raft, 3 pods)	every secret the cluster syncs through ESO	rebuild every credential by hand
etcd on the 3 controllers	the cluster state, encrypted Secrets	rebuild the cluster from git
VictoriaMetrics, VictoriaLogs	30 and 14 days of history	a gap in the graphs

Everything else is in git, and Flux puts it back.

Why not Velero

Velero is the usual answer, and I decided against it. The manifests are already in git and the secrets are in OpenBao. So a Velero backup of the objects is a third copy of things I already have. What is left is data inside running databases. A file-level copy of a Postgres volume while Postgres writes to it is neither consistent nor point-in-time. Each store has its own backup tool that does this correctly, so each store gets that tool.

flowchart LR
  PG["CNPG pg"] -->|"WAL, continuous"| R2["R2 bucket"]
  PG -->|"base backup, 12:00"| R2
  BAO["OpenBao Raft"] -->|"snapshot, 12:30"| R2
  K0S["k0s controller-01"] -->|"age-encrypted, 13:00"| R2
  R2 -.->|"restore test"| T["throwaway copies"]

The cluster is off from 23:30 to 06:00 every night (Part 10), so every schedule sits in the middle of the day.

Setup

The bucket

One R2 bucket, homelab-backups, with prefixes pg/, openbao/ and k0s/. R2 has no egress fees, and that matters for restores. The free tier covers 10 GB. The token is scoped to that one bucket with Object Read & Write:

GET /homelab-backups?list-type=2   200
GET /?x-id=ListBuckets             403

It cannot even list the other buckets in the account. The token sits in OpenBao twice, at secret/databases/r2-backup and secret/openbao/r2-backup. The reason is a Kyverno policy from Part 11: it only lets an ExternalSecret read the path of its own namespace.

Postgres: the Barman Cloud plugin

CNPG 1.30 still has the in-tree barmanObjectStore, but it is deprecated in favor of the Barman Cloud plugin. The plugin runs as a sidecar in each instance pod. I installed chart 0.8.1 (plugin v0.15.1) in cnpg-system after I compared its digest with the repository index. The store:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: r2
spec:
  retentionPolicy: "14d"
  configuration:
    destinationPath: s3://homelab-backups/pg/
    endpointURL: https://<account id>.r2.cloudflarestorage.com
    s3Credentials:
      accessKeyId: {name: r2-backup, key: access_key_id}
      secretAccessKey: {name: r2-backup, key: secret_access_key}
    wal:
      compression: gzip
    data:
      compression: gzip
  instanceSidecarConfiguration:
    env:
      # boto3 >= 1.36 sends checksums some S3-compatible stores reject
      - {name: AWS_REQUEST_CHECKSUM_CALCULATION, value: when_required}
      - {name: AWS_RESPONSE_CHECKSUM_VALIDATION, value: when_required}

The cluster points at it. That makes the plugin the WAL archiver:

spec:
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: r2

And a daily base backup. CNPG’s cron has a seconds field and runs in UTC, so 12:00 WIB is 0 0 5 * * *:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: pg-daily
spec:
  schedule: "0 0 5 * * *"
  immediate: true
  cluster:
    name: pg
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Adding the plugin restarts both instances once (the sidecar is new). This causes one switchover. LiteLLM logged “key lookup did not answer within 10s” during it, and its request-error alert fired.

OpenBao: a Raft snapshot CronJob

OpenBao has no backup operator, but bao operator raft snapshot save writes a consistent snapshot of the whole store. The CronJob runs it at 12:30 WIB in two containers:

snapshot (the OpenBao image) logs in with a Kubernetes auth role, raft-snapshot, whose policy allows exactly one thing:
```
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}
```
It writes the snapshot to an emptyDir. The pod gets only ca.crt from the OpenBao TLS secret, not the private key.
upload (rclone) copies it to r2:homelab-backups/openbao/ and pings a healthchecks.io check (/fail on error).

OpenBao’s barrier (its storage encryption layer) encrypts the snapshot. So the copy in R2 is useless without the unseal key, and that key is not in R2. A bucket lifecycle rule deletes snapshots after 30 days.

etcd: an encrypted `k0s backup`

The k0s controllers are not Kubernetes nodes, so this one is a systemd timer on k0s-controller-01, not a CronJob. k0s backup writes the etcd snapshot, the PKI and k0s.yaml. I add /etc/k0s/encryption-config.yaml (the Secret encryption key from Part 26), because the etcd snapshot is useless without it.

That bundle holds the private key of the cluster CA, so it never leaves the host unencrypted. The script encrypts it with age to a public key. The private half of that key exists only on my laptop:

k0s backup --data-dir /opt/lib/k0s --save-path "$work"
tar -cf - -C "$work" "$tarball" -C /etc/k0s encryption-config.yaml \
  | age -r "$AGE_RECIPIENT" > "$out"
curl -fsS --aws-sigv4 "aws:amz:auto:s3" \
  --user "$R2_ACCESS_KEY_ID:$R2_SECRET_ACCESS_KEY" \
  -T "$out" "$R2_ENDPOINT/$R2_BUCKET/k0s/$(basename "$out")"

curl --aws-sigv4 signs the upload, so the host needs no S3 client. The timer runs at 13:00 WIB with Persistent=true. If the VM was off at that time, the missed run happens after the next boot.

Issue 1: Kyverno blocked the first base backup

Symptom

The WAL archiver worked at once (about 4 seconds per segment). The immediate base backup failed:

error: unable to upgrade connection: admission webhook
"vpol.validate.kyverno.svc-ignore-43ea372b" denied the request:
Policy protect-sensitive-exec failed: exec/attach/port-forward into databases
is limited to cluster admins

Cause

My own policy from Part 11. Only system:masters can exec into the openbao, databases and cert-manager namespaces. So a compromised ServiceAccount with pods/exec cannot get a shell next to the data. The CNPG operator starts a plugin base backup by running /controller/manager backup <name> in the instance pod, and that is a pods/exec.

Fix

Allow the operator’s ServiceAccount, and only in databases:

validations:
  - expression: >-
      "system:masters" in request.userInfo.groups ||
      (request.namespace == "databases" &&
       request.userInfo.username == "system:serviceaccount:cnpg-system:cnpg-cloudnative-pg")

A manual backup right after took 16 seconds.

Issue 2: Tetragon flagged every backup, and the first fix did nothing

Symptom

The “Tetragon policy hit” alert fired for unexpected-egress (the exfiltration detector from Part 11). It fired on every barman call from databases, and on the healthchecks ping of the snapshot job from openbao:

policy=unexpected-egress ns=databases bin=/venv/bin/barman-cloud-wal-archive n=5
policy=unexpected-egress ns=databases bin=/venv/bin/barman-cloud-wal-restore n=38
policy=unexpected-egress ns=openbao   bin=/usr/bin/wget n=1

The policy flags any TCP connect from those namespaces to an address outside the cluster and LAN ranges. R2 is on Cloudflare (172.64.x.x).

First fix

Give the Cloudflare ranges to the barman binaries only, with a second selector. Tetragon then still flags any other process in a database pod:

selectors:
- matchBinaries:
  - operator: NotPrefix
    values: [/venv/bin/barman-cloud-]
  matchArgs: [{index: 0, operator: NotDAddr, values: [<cluster and LAN>]}]
- matchBinaries:
  - operator: Prefix
    values: [/venv/bin/barman-cloud-]
  matchArgs: [{index: 0, operator: NotDAddr, values: [<cluster, LAN, Cloudflare>]}]

Tetragon reloaded it at 14:03:00 UTC. The next WAL archive, at 14:03:46, fired again, and so did the one at 14:08:47. The destination was 172.64.190.1, inside the allowed 172.64.0.0/13.

Cause

The event said binary: /venv/bin/barman-cloud-wal-archive, but the argument list started with that same path. The barman tools are Python scripts. The kernel runs the interpreter from the shebang. Tetragon’s in-kernel matchBinaries compares against the file that executes: /venv/bin/python3. In the distroless sidecar, that path resolves to /usr/bin/python3.13. The event shows the script path for humans. The match uses the interpreter.

Fix

Add the interpreter as a second prefix in both selectors:

values: [/venv/bin/barman-cloud-, /usr/bin/python3]

Zero hits after the next archive. The OpenBao snapshot job got a podSelector exception instead (label app=raft-snapshot). The healthchecks.io servers sit on hosting IPs that change, and the snapshot that the job uploads is barrier-encrypted anyway.

Issue 3: The first restore failed three times

Symptom

A backup only counts after a successful restore, so I created a throwaway cluster from R2:

bootstrap:
  recovery:
    source: origin
externalClusters:
  - name: origin
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters: {barmanObjectName: r2, serverName: pg}

It went healthy after 4 minutes 21 seconds, but only after three recovery pods died with:

encountered an error while checking the presence of first needed WAL

Cause

Timing, not a broken backup. The base backup finished at 13:45:51. To be consistent, it needs the WAL segment that Postgres was writing when it ended. Postgres archives a segment only when it switches to the next one. On a quiet database that happens at archive_timeout, about 5 minutes. The first recovery pods started before that segment was in R2. The fourth found it.

Result

The restored litellm database matched the live one exactly:

pg-2:              28 spend rows | 1 key | 13 MB
pg-restore-test-1: 28 spend rows | 1 key | 13 MB

Restoring OpenBao and k0s

OpenBao. I started a single-node OpenBao in a scratch pod. It had its own Raft storage, a listener on localhost only, no retry_join, and the same static seal key. After bao operator init and bao operator raft snapshot restore -force, it unsealed itself with the static key. bao status showed the same cluster ID as the real one. All 11 KV namespaces were there. The token that I used that evening (it was in the snapshot) read a secret written an hour earlier.

k0s. I downloaded the object to the laptop, decrypted it with the age key, and listed it: etcd-snapshot.db, k0s.yaml, the k0s manifests, the PKI (29 files plus 8 for etcd) and encryption-config.yaml. I did not restore etcd into a scratch controller. k0s restore on a fresh VM is the next test.

Result

Backup	Schedule	Size and time	Restore test
Postgres WAL	every segment	about 4 s each	part of PITR
Postgres base	daily 12:00 WIB	16 s	healthy in 4m21s, data identical
OpenBao snapshot	daily 12:30 WIB	56 KB, upload 1.9 s	same cluster ID, all secrets readable
k0s backup	daily 13:00 WIB	79 MB, 13 s	decrypted and listed, not restored yet

Retention: 14 days of Postgres recovery (the plugin deletes older backups), 30 days of snapshots (bucket lifecycle rules). Everything fits the R2 free tier.

Knowing it still works

A backup that stopped without a warning is worse than none, because you rely on it.

Postgres: Grafana alerts when the last base backup is older than 26 hours, when a backup fails, and when more than 5 WAL segments wait for the archiver for 15 minutes.
OpenBao: the CronJob pings a healthchecks.io check on success and /fail on error. The check expects a ping daily at 12:30 with 2 hours of grace. So it also catches a run that never starts.
Grafana itself: an always-firing Watchdog rule pings another check every two minutes. If the lab, Grafana or VictoriaMetrics dies, the email comes from outside the house.

Lessons

Replicas are not backups. Two instances and a three-pod Raft cluster replicate a mistake as fast as they replicate data.
Use each store’s own backup tool. WAL archiving and Raft snapshots are consistent. A file copy of a running database volume is not.
Your own guardrails will block the backup. Both blocks came from my policies: Kyverno’s exec rule and Tetragon’s egress detector. Make each exception as narrow as the tool allows.
Tetragon matches the interpreter, not the script. For Python or shell tools, matchBinaries needs the interpreter’s real path, even though the event shows the script.
A restore right after a backup waits for one WAL switch. Retry, or force one with pg_switch_wal().
Encrypt what holds keys. The k0s backup carries the cluster CA, so it is encrypted to a key that never touches the cluster.
It is not a backup until you restore it. Each test here found something: the WAL timing, and that the OpenBao snapshot needs the seal key next to it.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 31: First Backups, Postgres, OpenBao and etcd to R2

Intro

The Figure

The Situation

Why not Velero

Setup

The bucket

Postgres: the Barman Cloud plugin

OpenBao: a Raft snapshot CronJob

etcd: an encrypted k0s backup

Issue 1: Kyverno blocked the first base backup

Symptom

Cause

Fix

Issue 2: Tetragon flagged every backup, and the first fix did nothing

Symptom

First fix

Cause

Fix

Issue 3: The first restore failed three times

Symptom

Cause

Result

Restoring OpenBao and k0s

Result

Knowing it still works

Lessons

Series

References

etcd: an encrypted `k0s backup`