Back to Home

Homelab Rebuild Part 2: Starting Over When the IaC Is Broken

| October 05, 2026 | 7 min to read

Intro

The three Proxmox hosts were now on Proxmox VE 9 (Part 1). But the old k0s cluster was still in a bad state. It had thirteen VMs and a Terraform config that no longer matched reality. I had also deleted the self-hosted GitHub runner by accident, so CI could not plan or apply anything.

I decided to start over: delete the old VMs, rebuild the cluster by hand, and bring it back under IaC later.

TL;DR: When your IaC is broken, rebuild by hand, but write every step down as a script. Then importing it back into Terraform later is mechanical. And keep the new layout small: 3 controllers and 5 workers instead of 13 VMs.

The Situation

Hardware

HostCPURAMDisks
pve-018 cores30 GBSamsung 990 PRO 1TB NVMe, Samsung 870 EVO 1TB SATA SSD
pve-028 cores30 GBsame as pve-01
pve-038 cores7.6 GB128 GB SATA SSD, 1TB 5400rpm laptop HDD

pve-03 is the odd one out. It matters later (Part 8): in the end it runs only workloads that need a quorum member, and nothing else.

Software

  • Proxmox VE 9.2, kernel 7.0
  • Debian 13 (trixie) genericcloud image for the VMs
  • k0s v1.36.4 installed with k0sctl 0.33.1
  • Tailscale on every VM for remote access

Setup

The layout

The old cluster had five workers per big host. I cut it down to what I need:

RoleCountPlacementSize
Controller3one per host2 vCPU, 3 GB (2 GB on pve-03), 40 GB
Worker2 + 2 + 1two on pve-01, two on pve-02, one on pve-034 vCPU, 8 GB, 100 GB

The API is behind a keepalived VIP (192.168.101.151), a virtual IP that moves between controllers. k0s manages it itself, so any controller can serve the API.

A Debian 13 template per node

Each Proxmox node gets its own template. VMIDs are cluster-wide, and full clones are fastest from local storage:

wget -N https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2
wget -O SHA512SUMS https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS
grep " debian-13-genericcloud-amd64.qcow2$" SHA512SUMS | sha512sum -c -

qm create 9100 --name debian13-cloudinit --memory 1024 --cores 2 --cpu host --ostype l26 \
  --machine q35 --bios seabios --scsihw virtio-scsi-single \
  --net0 virtio,bridge=vmbr0,firewall=1 --agent enabled=1 --serial0 socket --vga serial0
qm set 9100 --scsi0 local-lvm:0,import-from=/root/debian-13-genericcloud-amd64.qcow2,discard=on,ssd=1,iothread=1 \
  --ide2 local-lvm:cloudinit --boot order=scsi0
qm template 9100

The script checks against the published SHA512SUMS, not a hard-coded hash. So it keeps working when latest moves.

A cloud-init vendor snippet

Everything that every VM needs goes into one vendor snippet in /var/lib/vz/snippets/. A vendor snippet is extra cloud-init config that Proxmox adds to each VM. It holds the guest agent, Tailscale from the trixie repo, and a full upgrade. It grew over the series. By Part 3 it also sets a sysctl that fixes a Cilium problem, and by Part 6 it installs my private root CA.

#cloud-config
package_update: true
package_upgrade: true
packages:
  - qemu-guest-agent
  - curl
  - gnupg
runcmd:
  - systemctl enable --now qemu-guest-agent
  - curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.noarmor.gpg -o /usr/share/keyrings/tailscale-archive-keyring.gpg
  - curl -fsSL https://pkgs.tailscale.com/stable/debian/trixie.tailscale-keyring.list -o /etc/apt/sources.list.d/tailscale.list
  - apt-get update
  - apt-get install -y tailscale

Cloning the VMs

Each VM is a full clone with a static IP and SSH keys only. The old setup had a root password committed in git. This one has none.

qm clone 9100 101 --name k0s-pve-01-worker-1 --full 1 --storage local-lvm
qm set 101 --cores 4 --memory 8192 --onboot 1 --ciuser root --sshkeys /root/keys.pub \
  --ipconfig0 ip=192.168.101.40/24,gw=192.168.101.200 --nameserver '192.168.101.200 1.1.1.3' \
  --cicustom vendor=local:snippets/debian13-k0s.yml --ciupgrade 1
qm disk resize 101 scsi0 100G
qm start 101

Cloud-init with a full upgrade takes a few minutes per VM. Eight VMs that pull packages at the same time are the slow part of the whole rebuild.

Issue 1: My Mac Could Not Reach Its Own LAN

Symptom

k0sctl apply failed on every host:

ssh dial: dial tcp 192.168.101.30:22: connect: no route to host

But ssh root@192.168.101.30 from the same terminal worked.

Cause

On this Mac, only Apple-signed binaries (/usr/bin/ssh, /usr/bin/nc, /usr/bin/curl) could reach 192.168.101.0/24. The Homebrew nc, kubectl and k0sctl all failed. Tailscale addresses (100.x) worked for every binary. I never found the exact macOS cause. It was not the terminal app, a stale AdGuard extension or my sandbox.

Fix

Route around it through Tailscale. The Proxmox node pve-01 resolves to its Tailscale IP, so k0sctl can use it as an SSH bastion:

ssh:
  address: 192.168.101.40
  user: root
  keyPath: ~/.ssh/id_ecdsa
  bastion:
    address: pve-01
    user: root
    keyPath: ~/.ssh/id_rsa

For kubectl, the long-term answer is the Tailscale operator (Part 4).

Issue 2: k0sctl Rolled Back a Healthy Install

Symptom

The first real apply went all the way through. Then it failed and cleaned up:

failed on 5 hosts: context deadline exceeded
node k0s-pve-01-worker-1 is not ready

Cause

The cluster uses a custom CNI (provider: custom, kube-proxy disabled). So workers stay NotReady until Cilium runs. By default, k0sctl 0.33 waits for nodes to be Ready. It times out and rolls back.

Fix

k0sctl apply --config k0sctl.yaml --no-wait

Then install the CNI. That part became its own post (Part 3).

Lessons

  1. Script the manual path. Every qm command went into a script in the repo (proxmox/create-template.sh, proxmox/create-k0s-vm.sh). Later I imported the VMs back into Terraform (Part 7), and that was only a matter of copying values.
  2. Size the cluster for the hardware you have. Two strong hosts and one weak one is a different design problem than three equal ones.
  3. When a tool fails and a sibling tool works, compare binaries, not networks. The “no route to host” was about which program sent the packet, not about routing.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken (this post)
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths
  28. Part 28: Observability, Take Three: Making It Worth Looking At
  29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
  30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
  31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
  32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
  33. Part 33: The Firewall Held, the Tailnet Did Not
  34. Part 34: From Watching to Blocking, and a Record of Who Did What
  35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
  36. Part 36: The Fans Were Loud, the CPUs Were Idle
  37. Part 37: A Power Trip, a Search Domain and a Wildcard
  38. Part 38: An Autonomous AI Pentester in a Sealed Box
  39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References