Homelab Rebuild Part 8: Stateful Workloads on Unequal Hardware

Intro

The cluster needed a database. My first idea was CloudNativePG on Longhorn. Then I looked at what the hardware actually offered, and at what each layer already replicates.

TL;DR: Label each worker with its physical host (topology.kubernetes.io/zone), taint the weak host’s worker so only quorum members run there, give one worker per strong host a 100G NVMe data disk, and run CloudNativePG with 2 instances on node-local storage. PostgreSQL already replicates itself, so Longhorn underneath would only copy every write twice.

The Situation

• CloudNativePG 1.30.1 (chart 0.29.1), PostgreSQL 18.6
• local-path-provisioner v0.0.37

Setup

Topology: two workers per host is a trap

Each strong host runs two workers. Anti-affinity on node names alone happily puts both “redundant” replicas on the same physical machine. So every worker gets a zone label for its host, persisted in k0sctl.yaml:

installFlags:
  - --labels=topology.kubernetes.io/zone=pve-01,lab/data-disk=true

pve-03: quorum only

pve-03 had 0 GB free, and its worker used about 1 GB of its 4 GB. I trimmed it to 2.5 GB (drain, qm set --memory 2560, reboot, uncordon), which gave the host 3 GB back, and tainted it:

installFlags:
  - --taints=dedicated=quorum:NoSchedule

Only OpenBao tolerates the taint, because its third Raft voter has to live on a third machine for quorum to survive a host failure. Everything else runs on the two fast hosts. DaemonSets like Cilium tolerate all taints, so networking keeps running there.

NVMe data disks

One worker per strong host gets a 100G disk from the NVMe pool, hot-plugged without a reboot, formatted ext4, mounted at /mnt/data by UUID. local-path then sends those nodes there:

{"nodePathMap": [
  {"node": "DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths": ["/opt/local-path-provisioner"]},
  {"node": "k0s-pve-01-worker-1", "paths": ["/mnt/data/local-path"]},
  {"node": "k0s-pve-02-worker-1", "paths": ["/mnt/data/local-path"]}
]}

CloudNativePG without Longhorn

CloudNativePG replicates at the PostgreSQL level: a primary plus streaming replicas, each with its own volume. Its own documentation recommends node-local storage for that reason. Longhorn below it would add network round trips to every write and double the disk use, with no extra safety.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg
spec:
  instances: 2
  storage:
    storageClass: local-path
    size: 20Gi
  affinity:
    nodeSelector:
      lab/data-disk: "true"
    enablePodAntiAffinity: true
    podAntiAffinityType: required
    topologyKey: topology.kubernetes.io/zone

A row written on pg-1 was readable on pg-2 a second later. Backups to object storage come later; replicas don’t protect against a bad DROP TABLE.

Issue 1: mkfs Matched Two Disks

Symptom

To format the new data disk, my first script picked it by size:

d=$(lsblk -dnpo NAME,SIZE,TYPE | awk '$2=="100G" && $3=="disk" {print $1}')
mkfs.ext4 -q -L data "$d"

It failed:

The file /dev/sda
/dev/sdb does not exist and no size was specified.

Cause

The root disk was also 100G, so $d held both devices. The newline between them made mkfs see one nonexistent path, which is the only reason nothing was formatted. The device names were also swapped between the two VMs: on one the new disk was sdb, on the other sda, with root on sdb.

Fix

Both root filesystems were intact. The safe version selects by state, not by size or name: a whole disk with no partitions, no filesystem and no mountpoint, and it stops unless exactly one disk matches:

for d in $(lsblk -dnpo NAME,TYPE | awk '$2=="disk"{print $1}'); do
  [ "$(lsblk -npo NAME "$d" | wc -l)" = 1 ] || continue   # no partitions
  [ -z "$(lsblk -dno FSTYPE "$d")" ] || continue          # no filesystem
  [ -z "$(lsblk -dno MOUNTPOINT "$d")" ] || continue      # not mounted
  c="$c $d"
done
set -- $c; [ $# -eq 1 ] || { echo "expected 1 blank disk, found: $c"; exit 1; }

Issue 2: The First Volume Landed on the Root Disk

Symptom

pg-1’s volume was under /opt/local-path-provisioner, pg-2’s under /mnt/data.

Cause

The provisioner hadn’t reloaded its new nodePathMap yet when pg-1’s volume was created.

Fix

The cluster was empty, so I deleted it with its volumes and recreated it. Both volumes then landed on the NVMe disks.

Lessons

1. Don’t stack replication layers. If the application replicates, give it local disks.
2. Topology labels should describe physical failure domains, not VMs.
3. A weak node can still carry its weight as the tiebreaker for quorum.
4. Select disks by state, never by size or device name. Both can collide or change between otherwise identical VMs.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware (this post)
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers

References

• CloudNativePG: Storage
• CloudNativePG: Scheduling
• Rancher: local-path-provisioner