Loading

Back to Home

Homelab Rebuild Part 7: Self-Hosted CI and Getting Back to IaC

Technology Terraform Homelab Ci
| October 10, 2026 | 4 min to read

Intro

Part 2 rebuilt the VMs by hand because the Terraform setup was broken: its CI needed a self-hosted runner that no longer existed, and its state still listed thirteen VMs I had deleted. This part closes the loop: a runner inside the new cluster, a clean state, and the new VMs imported back into Terraform.

TL;DR: Actions Runner Controller 0.15.0 gives autoscaling runners (0 to 3) inside the cluster. The Terraform VM.Monitor error on Proxmox VE 9 is a provider bug, not a permission problem. removed blocks with destroy = false dropped the old VMs from state without destroying seven live VMs that reused their VMIDs. Finally, import blocks brought the eight new VMs into bpg/proxmox 0.116.0 with 8 imported, 0 changed.

The Situation

  • GitHub org repo with Terraform Plan and Terraform Apply workflows that run on [self-hosted, linux, x64, homelab], runner group homelab
  • AWS credentials through GitHub OIDC, state in S3
  • Telmate/proxmox 3.0.2-rc03, a Proxmox API token per workflow

Setup

Autoscaling runners in the cluster

ARC runner scale sets now support custom labels, so the existing workflows work unchanged:

githubConfigUrl: https://github.com/<org>
githubConfigSecret: arc-github-app
runnerScaleSetName: homelab
runnerGroup: homelab
scaleSetLabels: [self-hosted, linux, x64, homelab]
minRunners: 0
maxRunners: 3

Authentication is a GitHub App with the org permission “Self-hosted runners: Read and write”. Before creating the Secret, I minted an installation token from the key to check the permissions:

installation account: <org> | repo selection: selected
permissions: {'organization_self_hosted_runners': 'write', ...}

A Plan job that had been queued for three hours got picked up within seconds: a runner pod appeared, ran the job, and the scale set went back to zero.

Issue 1: The VM.Monitor Error

Symptom

permissions for user/token terraform-plan@pve are not sufficient,
please provide also the following permissions that are missing: [VM.Monitor]

The user already had Administrator on /.

Cause

Proxmox VE 9 removed the VM.Monitor privilege. Not even Administrator has it anymore. Telmate/proxmox 3.0.2-rc03 still checks for it.

Fix

Bump the provider: 3.0.2-rc04 dropped the check. Adding permissions can’t fix a privilege that no longer exists.

Issue 2: A Cleanup Plan That Wanted to Destroy the New Cluster

Symptom

The PR that deleted the old VM resources planned this:

Plan: 0 to add, 0 to change, 7 to destroy.
  # proxmox_vm_qemu.controller["controller-1"] will be destroyed

The old VMs were already gone. What would it destroy?

Cause

The rebuilt VMs reused seven of the old VMIDs (200, 300, 400, 101, 102, 201, 202). During refresh, Terraform found live VMs under those IDs and treated them as its own. Removing the resource blocks meant destroying them.

Fix

Forget the resources instead of destroying them:

removed {
  from = proxmox_vm_qemu.controller
  lifecycle {
    destroy = false
  }
}

The plan became 0 to destroy, and the apply only cleaned the state.

Issue 3: Importing Hand-Built VMs

Symptom

Telmate’s import fidelity is poor: attributes it can’t read back can force a replacement of a running node.

Fix

I switched to bpg/proxmox 0.116.0, wrote one for_each resource from the live qm config values (MACs, disks, cloud-init, tags), and used import blocks, which accept for_each since Terraform 1.7:

import {
  for_each = local.k0s_vms
  to       = proxmox_virtual_environment_vm.k0s[each.key]
  id       = "${each.value.node}/${each.value.vm_id}"
}

prevent_destroy guards every VM. CI builds bpg’s PROXMOX_VE_API_TOKEN from the existing secrets, so no new secret was needed. The first plan was clean:

Plan: 8 to import, 0 to add, 0 to change, 0 to destroy.
Apply complete! Resources: 8 imported, 0 added, 0 changed, 0 destroyed.

Lessons

  1. Read the plan, every line. A cleanup PR is exactly where a destroy hides.
  2. Reused IDs make state dangerous. Terraform binds resources by ID, not by intent.
  3. An error naming a permission isn’t always about permissions. Check whether the permission still exists.
  4. Import with import blocks in a PR. The plan shows what the import would change before anything happens.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC (this post)
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers

References