Intro
The three Proxmox hosts were now on Proxmox VE 9 (Part 1). But the old k0s cluster was still in a bad state. It had thirteen VMs and a Terraform config that no longer matched reality. I had also deleted the self-hosted GitHub runner by accident, so CI could not plan or apply anything.
I decided to start over: delete the old VMs, rebuild the cluster by hand, and bring it back under IaC later.
TL;DR: When your IaC is broken, rebuild by hand, but write every step down as a script. Then importing it back into Terraform later is mechanical. And keep the new layout small: 3 controllers and 5 workers instead of 13 VMs.
The Situation
Hardware
| Host | CPU | RAM | Disks |
|---|---|---|---|
pve-01 | 8 cores | 30 GB | Samsung 990 PRO 1TB NVMe, Samsung 870 EVO 1TB SATA SSD |
pve-02 | 8 cores | 30 GB | same as pve-01 |
pve-03 | 8 cores | 7.6 GB | 128 GB SATA SSD, 1TB 5400rpm laptop HDD |
pve-03 is the odd one out. It matters later (Part 8): in the end it runs only
workloads that need a quorum member, and nothing else.
Software
- Proxmox VE 9.2, kernel 7.0
- Debian 13 (trixie) genericcloud image for the VMs
- k0s v1.36.4 installed with k0sctl 0.33.1
- Tailscale on every VM for remote access
Setup
The layout
The old cluster had five workers per big host. I cut it down to what I need:
| Role | Count | Placement | Size |
|---|---|---|---|
| Controller | 3 | one per host | 2 vCPU, 3 GB (2 GB on pve-03), 40 GB |
| Worker | 2 + 2 + 1 | two on pve-01, two on pve-02, one on pve-03 | 4 vCPU, 8 GB, 100 GB |
The API is behind a keepalived VIP (192.168.101.151), a virtual IP that moves
between controllers. k0s manages it itself, so any controller can serve the API.
A Debian 13 template per node
Each Proxmox node gets its own template. VMIDs are cluster-wide, and full clones are fastest from local storage:
wget -N https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2
wget -O SHA512SUMS https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS
grep " debian-13-genericcloud-amd64.qcow2$" SHA512SUMS | sha512sum -c -
qm create 9100 --name debian13-cloudinit --memory 1024 --cores 2 --cpu host --ostype l26 \
--machine q35 --bios seabios --scsihw virtio-scsi-single \
--net0 virtio,bridge=vmbr0,firewall=1 --agent enabled=1 --serial0 socket --vga serial0
qm set 9100 --scsi0 local-lvm:0,import-from=/root/debian-13-genericcloud-amd64.qcow2,discard=on,ssd=1,iothread=1 \
--ide2 local-lvm:cloudinit --boot order=scsi0
qm template 9100
The script checks against the published SHA512SUMS, not a hard-coded hash.
So it keeps working when latest moves.
A cloud-init vendor snippet
Everything that every VM needs goes into one vendor snippet in
/var/lib/vz/snippets/. A vendor snippet is extra cloud-init config that Proxmox
adds to each VM. It holds the guest agent, Tailscale from the trixie repo, and a
full upgrade. It grew over the series. By Part 3 it also sets a sysctl that fixes
a Cilium problem, and by Part 6 it installs my private root CA.
#cloud-config
package_update: true
package_upgrade: true
packages:
- qemu-guest-agent
- curl
- gnupg
runcmd:
- systemctl enable --now qemu-guest-agent
- curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.noarmor.gpg -o /usr/share/keyrings/tailscale-archive-keyring.gpg
- curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.tailscale-keyring.list -o /etc/apt/sources.list.d/tailscale.list
- apt-get update
- apt-get install -y tailscale
Cloning the VMs
Each VM is a full clone with a static IP and SSH keys only. The old setup had a root password committed in git. This one has none.
qm clone 9100 101 --name k0s-pve-01-worker-1 --full 1 --storage local-lvm
qm set 101 --cores 4 --memory 8192 --onboot 1 --ciuser root --sshkeys /root/keys.pub \
--ipconfig0 ip=192.168.101.40/24,gw=192.168.101.200 --nameserver '192.168.101.200 1.1.1.3' \
--cicustom vendor=local:snippets/debian13-k0s.yml --ciupgrade 1
qm disk resize 101 scsi0 100G
qm start 101
Cloud-init with a full upgrade takes a few minutes per VM. Eight VMs that pull packages at the same time are the slow part of the whole rebuild.
Issue 1: My Mac Could Not Reach Its Own LAN
Symptom
k0sctl apply failed on every host:
ssh dial: dial tcp 192.168.101.30:22: connect: no route to host
But ssh root@192.168.101.30 from the same terminal worked.
Cause
On this Mac, only Apple-signed binaries (/usr/bin/ssh, /usr/bin/nc,
/usr/bin/curl) could reach 192.168.101.0/24. The Homebrew nc, kubectl
and k0sctl all failed. Tailscale addresses (100.x) worked for every
binary. I never found the exact macOS cause. It was not the terminal app,
a stale AdGuard extension or my sandbox.
Fix
Route around it through Tailscale. The Proxmox node pve-01 resolves to its
Tailscale IP, so k0sctl can use it as an SSH bastion:
ssh:
address: 192.168.101.40
user: root
keyPath: ~/.ssh/id_ecdsa
bastion:
address: pve-01
user: root
keyPath: ~/.ssh/id_rsa
For kubectl, the long-term answer is the Tailscale operator (Part 4).
Issue 2: k0sctl Rolled Back a Healthy Install
Symptom
The first real apply went all the way through. Then it failed and cleaned up:
failed on 5 hosts: context deadline exceeded
node k0s-pve-01-worker-1 is not ready
Cause
The cluster uses a custom CNI (provider: custom, kube-proxy disabled). So
workers stay NotReady until Cilium runs. By default, k0sctl 0.33 waits for
nodes to be Ready. It times out and rolls back.
Fix
k0sctl apply --config k0sctl.yaml --no-wait
Then install the CNI. That part became its own post (Part 3).
Lessons
- Script the manual path. Every
qmcommand went into a script in the repo (proxmox/create-template.sh,proxmox/create-k0s-vm.sh). Later I imported the VMs back into Terraform (Part 7), and that was only a matter of copying values. - Size the cluster for the hardware you have. Two strong hosts and one weak one is a different design problem than three equal ones.
- When a tool fails and a sibling tool works, compare binaries, not networks. The “no route to host” was about which program sent the packet, not about routing.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken (this post)
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed