Homelab Rebuild Part 22: Turning Four Days of Fixes into AI Runbooks

Intro

The rebuild so far: a Proxmox VE 8 to 9 upgrade, a new k0s cluster, FluxCD, OpenBao, and three hardware outages in four days. Every fix is in git, in a README, in a blog draft, or in Claude Code’s memory. But if I must start over, nothing tells me the order of the steps.

So I asked Claude Code to turn the work so far into skills: runbooks it loads on demand, one per operation. This post covers what it built and the two problems that came up during the writing. It also covers the change workflow I wrote down at the end.

TL;DR: Six thin runbooks in .claude/skills/. Each one is an ordered list of steps, health gates (checks that must pass before the next step) and gotchas. Each points at the existing READMEs instead of copying them. Writing them was a doc audit. It found a stale README and a reboot script that can take down OpenBao quorum, both Postgres instances and Pi-hole at once, so I deleted the script. My end-to-end change workflow now lives in CLAUDE.md, where every session loads it.

The Situation

• Infra repo: Terraform for the 8 k0s VMs, k0sctl.yaml, FluxCD apps under kubernetes/apps/, host scripts under proxmox/, and a README per app.
• Claude Code already had context from earlier sessions: about 20 memory files (one fact each), blog-drafts/ with facts from every session, and the transcripts of the first sessions.
• The repo’s CLAUDE.md already held conventions (Flux owns the cluster, resource rules, redaction rules for the blog). One hook, doc-reminder.sh, runs after edits under kubernetes/apps/, clusters/, terraform/ or proxmox/. It reminds me to keep the docs current.

Setup

Runbooks that point, not copy

The commands were already documented. A skill that copies them becomes a second copy that drifts. So each skill holds only what the docs do not:

• the order of steps and why it matters
• the gate to pass before the next step
• the gotchas from past sessions, with the date they happened
• pointers to the README or script that has the actual commands

Each skill is one Markdown file in the repo, versioned with the code it describes:

.claude/skills/
├── lab-pve-upgrade/SKILL.md
├── lab-vm-rebuild/SKILL.md
├── lab-cluster-bootstrap/SKILL.md
├── lab-controller-roll/SKILL.md
├── lab-add-app/SKILL.md
└── lab-outage-triage/SKILL.md

The front matter description decides when Claude loads a skill without a request from me. I can also call one directly as /lab-controller-roll.

---
name: lab-controller-roll
description: Roll a change across the 3 k0s controllers one at a time without losing etcd quorum (kube-apiserver flags, k0s config, RAM/CPU resize, k0s upgrade), and recover a thrashing controller.
---

The six skills

What the gates look like

The gates are the most useful part. Part 18’s RAM upgrade and Part 20’s authentication-config rollout both followed the same rule: change one controller, wait until the control plane is whole, then change the next one. The skill turns that into three checks:

kubectl get --raw='/readyz?verbose' | grep etcd     # [+]etcd ok
kubectl -n kube-system get lease | grep k0s-ctrl    # 3 leases, fresh renewTime
kubectl -n kube-system get ds konnectivity-agent \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | grep -o 'K0S_CONTROLLER_COUNT[^}]*'
# value 3

The third check exists because k0s rewrites the konnectivity DaemonSet every time the live controller count changes (Part 18). So a count of 2 means a controller is still missing, even when kubectl get nodes looks fine.

The outage skill gets the same treatment from Parts 10, 12, 13 and 21. It has a table of symptoms mapped to past causes (power adapter, POST hang fixed by a CMOS clear, a silent r8169 NIC stall, the laptop charger). It lists what normally degrades while a host is down, and it has the recovery checklist. That checklist includes the pg_switch_wal() fix for a Postgres standby that caught up but still refuses connections.

One gap, flagged

The exact dist-upgrade commands from the Part 1 upgrade were not in the session transcripts, because they ran detached. The upgrade skill rebuilds them from the playbook in memory: back up /etc, change the codename, simulate, run dist-upgrade with --force-confold under systemd-run, run the checker, reboot, wait for 3 votes. The skill also says to compare it with the official guide before the next major upgrade. I prefer a runbook that admits a gap to one that looks complete.

Issue 1: The Doc Was Already Wrong

Symptom

The Proxmox README’s layout table said controller-01 and controller-02 have 3 GiB and the data disks are 100G.

Cause

Part 18 raised both controllers to 4 GiB, and Part 12 grew the data disks to 250G. Both changes went into vms.tf, but nobody updated the README table. The doc hook only reminds me. It cannot check that the numbers match.

Fix

I corrected it in the same PR as the skills. To write a runbook, you read every doc it points at, so it doubles as a doc audit.

Issue 2: A Reboot Script That Can Break Quorum

Symptom

scripts/reboot-nodes is older than the rebuild. It still listed the old 13-node layout (five workers per host), and its plan was:

Workers will be rebooted concurrently, then controllers one by one

Cause

On the old cluster that was only rude. On the current one it is an outage:

The controller half waited only for SSH to come back, not for etcd to be healthy. So it can take a second controller down before the first one rejoins.

Fix

I started to rewrite it as one node at a time with the same gates as the skills. A check of the live node names hit two sandbox denials in a row. First the kubectl binary path was not readable. Then the Tailscale API proxy host was not in the allowed domains. At that point I asked myself whether the script was worth the fight. It was not. I reboot rarely, and the skills already cover controllers and hosts, so I deleted all 325 lines. I reboot workers by hand, one at a time. Between each one, I wait for OpenBao 3/3 and pg 2/2.

Writing Down the Workflow

The last question was whether I can give Claude a standing workflow, not only runbooks. Claude Code has several places for instructions, and they differ in how strictly they apply:

My rule of thumb now: things that must always happen go in CLAUDE.md. Procedures I start on purpose become skills. Anything that I must never skip becomes a hook.

The workflow I described was short:

1. Configure what I asked for.
2. Ship it through the layer’s owner: PR, merge, then Flux applies Kubernetes changes and the self-hosted GitHub runner applies Terraform for Proxmox. I do ad hoc host work over ssh with qm/pvenode, then sync it back into IaC.
3. Find out whether the blog already covers it. If not, write it up, one post per category.

Claude added the parts I do but never wrote down:

• Verify after apply. A merge is not the end. Wait for Flux Ready or a green Apply run, then make sure that the change actually works.
• Prove there is no drift. After an ssh/qm change, the PR’s Terraform Plan must show “No changes”.
• Fix forward. No hand kubectl apply or helm upgrade while Flux owns the app.
• Never let CI change two controllers. The Proxmox provider reboots changed VMs in parallel, and that loses etcd quorum (Part 18).
• No applies overnight. VMs are off from 23:30 to 06:00 WIB. A queued Apply waits for a runner that does not exist and blocks the next run (Part 12).
• Check both the drafts and the published posts, and add an update section to an existing post instead of writing a near-duplicate.

The blog step changed one thing. Claude opens the blog PR, but I review and merge it. Only one blog PR stays open at a time. New posts go onto that branch. This post is the first one written under the new rule.

Lessons

1. Runbooks point, they do not copy. The commands already live in READMEs and scripts. A runbook adds order, gates and gotchas, and stays short enough to keep current.
2. Writing a runbook is an audit. One stale README and one dangerous script came out of a single session. I only read everything the runbooks reference.
3. Gates matter more than commands. “One controller at a time” means nothing until “done” has a definition: etcd ok, 3 fresh leases, K0S_CONTROLLER_COUNT=3.
4. Delete what you will not maintain. A script that encodes a layout from two rebuilds ago is worse than no script, because it looks like it works.
5. Pick the mechanism by how strict it must be. Always-on rules, on-demand procedures and enforced checks belong in different places.
6. Untested runbooks are still drafts. I did not run any of these skills end to end yet. The first real rebuild will be their test.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks (this post)
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• Claude Code: Skills
• Claude Code: Memory and CLAUDE.md
• Claude Code: Hooks
• Claude Code: Subagents
• k0s: Control plane high availability