Homelab Rebuild Part 22: Turning Four Days of Fixes into AI Runbooks

Intro

The rebuild so far: a Proxmox VE 8 to 9 upgrade, a new k0s cluster, FluxCD, OpenBao, and three hardware outages in four days. Every fix is in git, in a README, in a blog draft, or in Claude Code’s memory, but nothing tells me what order to do things in if I ever have to start over.

So I asked Claude Code to turn the work so far into skills: runbooks it loads on demand, one per operation. This post covers what it built, the two problems that came up while writing them, and the change workflow I wrote down at the end.

TL;DR: Six thin runbooks in .claude/skills/, each one an ordered list of steps, health gates and gotchas that points at the existing READMEs instead of copying them. Writing them was a doc audit: it found a stale README and a reboot script that would have taken down OpenBao quorum, both Postgres instances and Pi-hole at once, so I deleted it. My end-to-end change workflow now lives in CLAUDE.md, where every session loads it.

The Situation

• Infra repo: Terraform for the 8 k0s VMs, k0sctl.yaml, FluxCD apps under kubernetes/apps/, host scripts under proxmox/, and a README per app.
• Claude Code already had context from earlier sessions: about 20 memory files (one fact each), blog-drafts/ with facts from every session, and the transcripts of the first sessions.
• The repo’s CLAUDE.md already held conventions (Flux owns the cluster, resource rules, redaction rules for the blog). One hook, doc-reminder.sh, reminds after edits under kubernetes/apps/, clusters/, terraform/ or proxmox/ to keep the docs current.

Setup

Runbooks that point, not copy

The commands were already documented. A skill that copies them becomes a second copy that drifts. So each skill holds only what the docs don’t:

• the order of steps and why it matters
• the gate to pass before the next step
• the gotchas from past sessions, with the date they happened
• pointers to the README or script that has the actual commands

Each skill is one Markdown file in the repo, versioned with the code it describes:

.claude/skills/
├── lab-pve-upgrade/SKILL.md
├── lab-vm-rebuild/SKILL.md
├── lab-cluster-bootstrap/SKILL.md
├── lab-controller-roll/SKILL.md
├── lab-add-app/SKILL.md
└── lab-outage-triage/SKILL.md

The front matter description decides when Claude loads a skill without being asked. I can also call one directly as /lab-controller-roll.

---
name: lab-controller-roll
description: Roll a change across the 3 k0s controllers one at a time without losing etcd quorum (kube-apiserver flags, k0s config, RAM/CPU resize, k0s upgrade), and recover a thrashing controller.
---

The six skills

What the gates look like

The most useful part is the gates, not the commands. Part 18’s RAM upgrade and Part 20’s authentication-config rollout both followed the same rule: change one controller, wait until the control plane is whole, then the next. The skill turns that into three checks:

kubectl get --raw='/readyz?verbose' | grep etcd     # [+]etcd ok
kubectl -n kube-system get lease | grep k0s-ctrl    # 3 leases, fresh renewTime
kubectl -n kube-system get ds konnectivity-agent \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | grep -o 'K0S_CONTROLLER_COUNT[^}]*'
# value 3

The third check exists because k0s rewrites the konnectivity DaemonSet every time the live controller count changes (Part 18). So a count of 2 means a controller is still missing, even when kubectl get nodes looks fine.

The outage skill gets the same treatment from Parts 10, 12, 13 and 21: a table of symptoms mapped to past causes (power adapter, POST hang fixed by a CMOS clear, a silent r8169 NIC stall, the laptop charger), what normally degrades while a host is down, and the recovery checklist. That checklist includes the pg_switch_wal() fix for a Postgres standby that has caught up but still refuses connections.

One gap, flagged

The exact dist-upgrade commands from the Part 1 upgrade weren’t in the session transcripts, because they ran detached. The upgrade skill rebuilds them from the playbook in memory (back up /etc, change the codename, simulate, run dist-upgrade with --force-confold under systemd-run, run the checker, reboot, wait for 3 votes) and says to check it against the official guide before the next major upgrade. A runbook that admits a gap beats one that looks complete.

Issue 1: The Doc Was Already Wrong

Symptom

The Proxmox README’s layout table said controller-01 and controller-02 have 3 GiB and the data disks are 100G.

Cause

Part 18 raised both controllers to 4 GiB, and Part 12 grew the data disks to 250G. Both changes went into vms.tf, but the README table wasn’t updated. The doc hook only reminds me; it can’t check that the numbers match.

Fix

Corrected in the same PR as the skills. Writing a runbook means reading every doc it points at, so it doubles as a doc audit.

Issue 2: A Reboot Script That Would Break Quorum

Symptom

scripts/reboot-nodes predates the rebuild. It still listed the old 13-node layout (five workers per host), and its plan was:

Workers will be rebooted concurrently, then controllers one by one

Cause

On the old cluster that was merely rude. On the current one it’s an outage:

The controller half only waited for SSH to come back, not for etcd to be healthy, so it could take a second controller down before the first had rejoined.

Fix

I started to rewrite it as one node at a time with the same gates as the skills. Checking the live node names hit two sandbox denials in a row: the kubectl binary path wasn’t readable, then the Tailscale API proxy host wasn’t in the allowed domains. At that point I asked myself whether the script was worth the fight. It wasn’t. I reboot rarely, and the skills already cover controllers and hosts. Deleted, 325 lines. Workers get rebooted by hand, one at a time, waiting for OpenBao 3/3 and pg 2/2 in between.

Writing Down the Workflow

The last question was whether I could give Claude a standing workflow, not just runbooks. Claude Code offers several places to put instructions, and they differ in how strictly they apply:

My rule of thumb now: things that should always happen go in CLAUDE.md, procedures I start on purpose become skills, and anything that must never be skipped becomes a hook.

The workflow I described was short:

1. Configure what I asked for.
2. Ship it through the layer’s owner: PR, merge, then Flux applies Kubernetes changes and the self-hosted GitHub runner applies Terraform for Proxmox. Ad hoc host work is done over ssh with qm/pvenode, then synced back into IaC.
3. Check whether the blog already covers it, and write it up if not, one post per category.

Claude added the parts I do but never wrote down:

• Verify after apply. Merging isn’t done. Flux Ready or a green Apply run, then check that the change actually works.
• Prove there is no drift. After an ssh/qm change, the PR’s Terraform Plan must show “No changes”.
• Fix forward. No hand kubectl apply or helm upgrade while Flux owns the app.
• Never let CI change two controllers. The Proxmox provider reboots changed VMs in parallel, which loses etcd quorum (Part 18).
• No applies overnight. VMs are off from 23:30 to 06:00 WIB, so a queued Apply waits for a runner that doesn’t exist and blocks the next run (Part 12).
• Check both the drafts and the published posts, and add an update section to an existing post instead of writing a near-duplicate.

The blog step changed one thing: Claude opens the blog PR, but I review and merge it, and only one blog PR stays open at a time. New posts go onto that branch. This post is the first one written under the new rule.

Lessons

1. Runbooks should point, not copy. The commands already live in READMEs and scripts. A runbook adds order, gates and gotchas, and stays short enough to keep current.
2. Writing a runbook is an audit. One stale README and one dangerous script came out of a single session, just from reading everything the runbooks reference.
3. Gates matter more than commands. “One controller at a time” means nothing until “done” has a definition: etcd ok, 3 fresh leases, K0S_CONTROLLER_COUNT=3.
4. Delete what you won’t maintain. A script that encodes a layout from two rebuilds ago is worse than no script, because it looks like it works.
5. Pick the mechanism by how strict it must be. Always-on rules, on-demand procedures and enforced checks belong in different places.
6. Untested runbooks are still drafts. None of these skills has been run end to end yet. The first real rebuild will be their test.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks (this post)
23. Part 23: microVM Sandboxes on Proxmox

References

• Claude Code: Skills
• Claude Code: Memory and CLAUDE.md
• Claude Code: Hooks
• Claude Code: Subagents
• k0s: Control plane high availability