Homelab Rebuild Part 2: Starting Over When the IaC Is Broken

Intro

With the three Proxmox hosts on Proxmox VE 9 (Part 1), the old k0s cluster was still in a bad state: thirteen VMs, a Terraform config that no longer matched reality, and a self-hosted GitHub runner I had deleted by accident, so CI couldn’t plan or apply anything.

I decided to start over: delete the old VMs, rebuild the cluster by hand, and bring it back under IaC later.

TL;DR: When your IaC is broken, rebuild by hand but write every step down as a script, so importing it back into Terraform later is mechanical. And keep the new layout small: 3 controllers and 5 workers instead of 13 VMs.

The Situation

Hardware

pve-03 is the odd one out. It matters later (Part 8): it ends up running only workloads that need a quorum member, nothing else.

Software

• Proxmox VE 9.2, kernel 7.0
• Debian 13 (trixie) genericcloud image for the VMs
• k0s v1.36.4 installed with k0sctl 0.33.1
• Tailscale on every VM for remote access

Setup

The layout

The old cluster had five workers per big host. I cut it down to what I actually need:

The API is behind a keepalived VIP (192.168.101.151) that k0s manages itself, so any controller can serve it.

A Debian 13 template per node

Each Proxmox node gets its own template, because VMIDs are cluster-wide and full clones are fastest from local storage:

wget -N https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2
wget -O SHA512SUMS https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS
grep " debian-13-genericcloud-amd64.qcow2$" SHA512SUMS | sha512sum -c -

qm create 9100 --name debian13-cloudinit --memory 1024 --cores 2 --cpu host --ostype l26 \
  --machine q35 --bios seabios --scsihw virtio-scsi-single \
  --net0 virtio,bridge=vmbr0,firewall=1 --agent enabled=1 --serial0 socket --vga serial0
qm set 9100 --scsi0 local-lvm:0,import-from=/root/debian-13-genericcloud-amd64.qcow2,discard=on,ssd=1,iothread=1 \
  --ide2 local-lvm:cloudinit --boot order=scsi0
qm template 9100

Checking against the published SHA512SUMS instead of a hard-coded hash keeps the script working when latest moves.

A cloud-init vendor snippet

Everything that every VM needs goes into one vendor snippet in /var/lib/vz/snippets/: the guest agent, Tailscale from the trixie repo, and a full upgrade. It grew over the series. By Part 6 it also installs my private root CA, and by Part 3 a sysctl that fixes a Cilium problem.

#cloud-config
package_update: true
package_upgrade: true
packages:
  - qemu-guest-agent
  - curl
  - gnupg
runcmd:
  - systemctl enable --now qemu-guest-agent
  - curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.noarmor.gpg -o /usr/share/keyrings/tailscale-archive-keyring.gpg
  - curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.tailscale-keyring.list -o /etc/apt/sources.list.d/tailscale.list
  - apt-get update
  - apt-get install -y tailscale

Cloning the VMs

Each VM is a full clone with a static IP and SSH keys only. The old setup had a root password committed in git; this one has none.

qm clone 9100 101 --name k0s-pve-01-worker-1 --full 1 --storage local-lvm
qm set 101 --cores 4 --memory 8192 --onboot 1 --ciuser root --sshkeys /root/keys.pub \
  --ipconfig0 ip=192.168.101.40/24,gw=192.168.101.200 --nameserver '192.168.101.200 1.1.1.3' \
  --cicustom vendor=local:snippets/debian13-k0s.yml --ciupgrade 1
qm disk resize 101 scsi0 100G
qm start 101

Cloud-init with a full upgrade takes a few minutes per VM. Eight VMs pulling packages at the same time is the slow part of the whole rebuild.

Issue 1: My Mac Couldn’t Reach Its Own LAN

Symptom

k0sctl apply failed on every host:

ssh dial: dial tcp 192.168.101.30:22: connect: no route to host

Yet ssh [email protected] from the same terminal worked.

Cause

On this Mac, only Apple-signed binaries (/usr/bin/ssh, /usr/bin/nc, /usr/bin/curl) could reach 192.168.101.0/24. Homebrew’s nc, kubectl and k0sctl all failed, while Tailscale addresses (100.x) worked for every binary. I never pinned down the exact macOS cause; it wasn’t the terminal app, a stale AdGuard extension or my sandbox.

Fix

Route around it through Tailscale. The Proxmox node pve-01 resolves to its Tailscale IP, so k0sctl can use it as an SSH bastion:

ssh:
  address: 192.168.101.40
  user: root
  keyPath: ~/.ssh/id_ecdsa
  bastion:
    address: pve-01
    user: root
    keyPath: ~/.ssh/id_rsa

For kubectl, the long-term answer is the Tailscale operator (Part 4).

Issue 2: k0sctl Rolled Back a Healthy Install

Symptom

The first real apply got all the way through, then failed and cleaned up:

failed on 5 hosts: context deadline exceeded
node k0s-pve-01-worker-1 is not ready

Cause

The cluster uses a custom CNI (provider: custom, kube-proxy disabled), so workers stay NotReady until Cilium runs. k0sctl 0.33 waits for nodes to be Ready by default, times out, and rolls back.

Fix

k0sctl apply --config k0sctl.yaml --no-wait

Then install the CNI. That part turned into its own post (Part 3).

Lessons

1. Script the manual path. Every qm command went into a script in the repo (proxmox/create-template.sh, proxmox/create-k0s-vm.sh). Importing the VMs back into Terraform later (Part 7) was a matter of copying values.
2. Size the cluster for the hardware you have. Two strong hosts and one weak one is a different design problem than three equal ones.
3. When a tool fails and a sibling tool works, compare binaries, not networks. The “no route to host” was about which program sent the packet, not about routing.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken (this post)
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers

References

• Debian: Cloud images
• Proxmox VE: Cloud-Init support
• k0s: k0sctl install
• Tailscale: Install on Debian