Homelab Rebuild Part 8: Stateful Workloads on Unequal Hardware

Technology Kubernetes Homelab Postgresql

| October 11, 2026 | 7 min to read

Intro

The cluster needed a database. My first idea was CloudNativePG on Longhorn. Then I looked at what the hardware actually offered, and at what each layer already replicates.

TL;DR: Label each worker with its physical host (topology.kubernetes.io/zone), taint the weak host’s worker so only quorum members run there, give one worker per strong host a 100G NVMe data disk, and run CloudNativePG with 2 instances on node-local storage. PostgreSQL already replicates itself, so Longhorn underneath would only copy every write twice.

The Figure

Pointer: right of centre is quorumHosts 01 to 03 (pve-01, pve-02, pve-03), one Raft voter weight eachEach Proxmox host is a zone and runs one OpenBao Raft voter, so each puts one equal weight on the left pan; the counterweight is one and a half votes, so any two voters tip the beam. pve-03 has one worker, tainted dedicated=quorum:NoSchedule, and its weight counts the same as the others. pg-1 (primary) and pg-2 (streaming replica) sit on the 100G NVMe disks of pve-01 and pve-02, one per zone, with no Longhorn underneath.

Each Proxmox host puts one weight on the left pan: its OpenBao Raft voter. The counterweight on the right is one and a half votes, so any two hosts tip the beam to quorum, and the weak pve-03 counts exactly as much as the strong ones. Click a host or press 1, 2 or 3 to take it down: its lift pin pushes the weight off the pan and the readout shows what is left of Raft and of CloudNativePG, whose two instances sit on the NVMe disks of pve-01 and pve-02. I built the figure with the anatomy agent skill.

The Situation

Host	Free disk	Free RAM	Workers
`pve-01`	762G NVMe, 917G SATA	11 GB	2
`pve-02`	775G NVMe, 935G SATA	11 GB	2
`pve-03`	59G SSD (+ 1TB slow HDD)	0 GB	1

CloudNativePG 1.30.1 (chart 0.29.1), PostgreSQL 18.6
local-path-provisioner v0.0.37

Setup

Topology: two workers per host is a trap

Each strong host runs two workers. Anti-affinity on node names alone happily puts both “redundant” replicas on the same physical machine. So every worker gets a zone label for its host, persisted in k0sctl.yaml:

installFlags:
  - --labels=topology.kubernetes.io/zone=pve-01,lab/data-disk=true

pve-03: quorum only

pve-03 had 0 GB free, and its worker used about 1 GB of its 4 GB. I trimmed it to 2.5 GB (drain, qm set --memory 2560, reboot, uncordon), which gave the host 3 GB back, and tainted it:

installFlags:
  - --taints=dedicated=quorum:NoSchedule

Only OpenBao tolerates the taint, because its third Raft voter has to live on a third machine for quorum to survive a host failure. Everything else runs on the two fast hosts. DaemonSets like Cilium tolerate all taints, so networking keeps running there.

NVMe data disks

One worker per strong host gets a 100G disk from the NVMe pool, hot-plugged without a reboot, formatted ext4, mounted at /mnt/data by UUID. local-path then sends those nodes there:

{"nodePathMap": [
  {"node": "DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths": ["/opt/local-path-provisioner"]},
  {"node": "k0s-pve-01-worker-1", "paths": ["/mnt/data/local-path"]},
  {"node": "k0s-pve-02-worker-1", "paths": ["/mnt/data/local-path"]}
]}

CloudNativePG without Longhorn

CloudNativePG replicates at the PostgreSQL level: a primary plus streaming replicas, each with its own volume. Its own documentation recommends node-local storage for that reason. Longhorn below it would add network round trips to every write and double the disk use, with no extra safety.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg
spec:
  instances: 2
  storage:
    storageClass: local-path
    size: 20Gi
  affinity:
    nodeSelector:
      lab/data-disk: "true"
    enablePodAntiAffinity: true
    podAntiAffinityType: required
    topologyKey: topology.kubernetes.io/zone

Instance	Node	Host	Role
`pg-1`	`k0s-pve-01-worker-1`	pve-01	primary
`pg-2`	`k0s-pve-02-worker-1`	pve-02	streaming replica

A row written on pg-1 was readable on pg-2 a second later. Backups to object storage come later; replicas don’t protect against a bad DROP TABLE.

Issue 1: mkfs Matched Two Disks

Symptom

To format the new data disk, my first script picked it by size:

d=$(lsblk -dnpo NAME,SIZE,TYPE | awk '$2=="100G" && $3=="disk" {print $1}')
mkfs.ext4 -q -L data "$d"

It failed:

The file /dev/sda
/dev/sdb does not exist and no size was specified.

Cause

The root disk was also 100G, so $d held both devices. The newline between them made mkfs see one nonexistent path, which is the only reason nothing was formatted. The device names were also swapped between the two VMs: on one the new disk was sdb, on the other sda, with root on sdb.

Fix

Both root filesystems were intact. The safe version selects by state, not by size or name: a whole disk with no partitions, no filesystem and no mountpoint, and it stops unless exactly one disk matches:

for d in $(lsblk -dnpo NAME,TYPE | awk '$2=="disk"{print $1}'); do
  [ "$(lsblk -npo NAME "$d" | wc -l)" = 1 ] || continue   # no partitions
  [ -z "$(lsblk -dno FSTYPE "$d")" ] || continue          # no filesystem
  [ -z "$(lsblk -dno MOUNTPOINT "$d")" ] || continue      # not mounted
  c="$c $d"
done
set -- $c; [ $# -eq 1 ] || { echo "expected 1 blank disk, found: $c"; exit 1; }

Issue 2: The First Volume Landed on the Root Disk

Symptom

pg-1’s volume was under /opt/local-path-provisioner, pg-2’s under /mnt/data.

Cause

The provisioner hadn’t reloaded its new nodePathMap yet when pg-1’s volume was created.

Fix

The cluster was empty, so I deleted it with its volumes and recreated it. Both volumes then landed on the NVMe disks.

Lessons

Don’t stack replication layers. If the application replicates, give it local disks.
Topology labels should describe physical failure domains, not VMs.
A weak node can still carry its weight as the tiebreaker for quorum.
Select disks by state, never by size or device name. Both can collide or change between otherwise identical VMs.

Series

This post is part of Homelab Rebuild:

Homelab Rebuild Part 8: Stateful Workloads on Unequal Hardware

Intro

The Figure

The Situation

Setup

Topology: two workers per host is a trap

pve-03: quorum only

NVMe data disks

CloudNativePG without Longhorn

Issue 1: mkfs Matched Two Disks

Symptom

Cause

Fix

Issue 2: The First Volume Landed on the Root Disk

Symptom

Cause

Fix

Lessons

Series

References