Homelab Rebuild Part 41: n8n with Claude Code Runners on the Subscription

Intro

I wanted one place to run small automations against the lab: a morning summary of what broke overnight, a watch on upstream releases of what I run, and later, agents that work on my own code. Cron jobs and shell scripts would do the first two, but every new one would mean another image, another CronJob, another set of secrets. A workflow engine with a UI, an API and retries is the better shape.

n8n is that engine. This post covers how it sits in the cluster, how its steps call language models (cheap ones through LiteLLM, Claude through headless Claude Code on my subscription), the two errors on the way, and what it runs today.

TL;DR: n8n 2.45.7 in namespace n8n, database on the CloudNativePG cluster, reachable on the tailnet and the LAN only, behind oauth2-proxy with Cloudflare Access and Google because n8n Community has no SSO. Simple steps call LiteLLM with a virtual key that n8n pre-fills into every OpenAI credential. Hard steps call claude-runner, a 170-line Node wrapper around claude -p, one Deployment per model tier, authenticated with a claude setup-token subscription token, never an API key. The first boot died to its own liveness probe, and a Kyverno policy I wrote two weeks ago had been silently blocking every External Secret update.

The Figure

I drew n8n as a Jacquard loom. The chain of six punched cards on the table at the right is one workflow, a card per node, and the clock on the head turns to the trigger hour to start it. The card under the reader lights and lifts its hook. A cheap step spins the side bobbin on the left (LiteLLM on Workers AI). A hard step throws a shuttle from one of the three boxes on the right: haiku has three small shuttles, sonnet two and opus one big one. When every shuttle in a box is already out, its red flag goes up (a 429), and the card waits a beat before n8n tries again. The cloth rolling onto the front beam and the three lamps (Telegram, email and GitHub, left to right) show what got delivered. Click the figure, or press 1, 2 or 3, to load the morning briefing, the release watch or the dvarapala review. Built with the anatomy skill.

The Situation

• k0s cluster on three Proxmox hosts, FluxCD owns every app (Part 16).
• CloudNativePG cluster pg, two instances on NVMe data disks (Part 8).
• Secrets in OpenBao, synced by External Secrets (Part 5).
• Cloudflare Access with Google as the OIDC issuer for OpenBao, Headlamp, LiteLLM and the Proxmox UI (Part 19).
• LiteLLM in front of Cloudflare Workers AI (Part 29).
• Kyverno policy protect-flux-managed in Deny mode since Part 34.

Setup

A login n8n does not have

n8n Community has no OIDC or SAML login: both need a paid licence. So n8n is never reached directly. oauth2-proxy sits in front of it, with the same Cloudflare Access plus Google pattern as everything else in the lab.

flowchart LR
  Me["Mac / phone"] -->|tailnet| TS["Tailscale Ingress"]
  Me -->|home LAN| GW["lab-gateway<br/>n8n.franzramadhan.lab"]
  TS --> OP["oauth2-proxy"]
  GW --> OP
  OP -->|OIDC| CFA["Cloudflare Access<br/>(Google)"]
  OP --> N8N["n8n"]
  N8N --> PG[("CNPG pg")]

• Tailnet: a Tailscale Ingress to the oauth2-proxy Service. LAN: an HTTPRoute on lab-gateway with a lab CA certificate. Nothing on the internet.
• The Access policy allows one email, and oauth2-proxy checks it again with an emails file and emailDomains: []. The chart default is ["*"], which would let any Google account through once Access let it in.
• No fixed redirect-url. With reverse-proxy: true the callback follows the host the login started on, so the cookie lands on the right one. Both callback URLs are registered in the Access app.
• A CiliumNetworkPolicy lets only oauth2-proxy (and monitoring) reach n8n, so nothing inside the cluster can skip the login.
• /webhook/* skips oauth2-proxy (skip-auth-route), so webhook triggers still work. They are only reachable from the tailnet and the LAN.

The result is two logins in a row: Google, then n8n’s own owner account. n8n’s session cookie lasts 7 days. I kept it that way on purpose. The alternative is an EXTERNAL_HOOK_FILES hook that trusts X-Forwarded-Email, which uses internal APIs and breaks on upgrades.

LiteLLM for the cheap steps

n8n gets its own LiteLLM virtual key, limited to smart, glm-4.7-flash and glm-5.3-flash, with a budget of 5 USD per 30 days. Rather than typing it into the UI, CREDENTIALS_OVERWRITE_DATA (templated by the ExternalSecret) pre-fills every n8n OpenAI credential:

{"openAiApi": {"apiKey": "<from OpenBao>", "url": "http://litellm.litellm.svc:4000/v1"}}

Creating an OpenAI credential in n8n now needs no typing, and the key never leaves OpenBao and the Secret.

Two things bit me in the first workflow. smart can route to a reasoning model, and with max_tokens 400 the reasoning used the whole budget and the answer came back empty (now 2000). And LiteLLM caches identical prompts, so a summary prompt that did not change between runs returned yesterday’s answer. The prompt now includes the current time.

Claude on the subscription, not an API key

For steps that need a frontier model I wanted Claude, billed to the Claude subscription I already pay for rather than per token. Claude Code can run headless with a subscription token: claude setup-token prints a long-lived OAuth token, and the CLI reads it from CLAUDE_CODE_OAUTH_TOKEN. Runs count against the plan’s usage limits, shared with my interactive use. --bare is not an option here: in bare mode authentication is strictly ANTHROPIC_API_KEY or apiKeyHelper.

claude-runner/server.mjs is a small HTTP wrapper with only Node’s standard library:

POST /run {prompt, system_prompt?, max_turns?, tools?, plugins?}
  -> claude -p --model <tier> --output-format json --max-turns N
       --no-session-persistence --strict-mcp-config --tools <list>
  -> Claude's JSON result (result, num_turns, total_cost_usd, ...)

The prompt goes in on stdin, every request gets a fresh scratch directory, and each tier is its own Deployment and Service:

• A busy tier answers 429 instead of queueing. n8n’s Retry On Fail handles it.
• Tools default to none. Allowed: Read, Write, Edit, Glob, Grep, WebSearch. Bash is refused: the token is in the process environment, and a shell could print it.
• Read-only root filesystem, non-root, all capabilities dropped, automountServiceAccountToken: false, a scratch emptyDir with a size limit.
• The CiliumNetworkPolicy lets only n8n call the runners. The runners reach api.anthropic.com, and the init container registry.npmjs.org and codeload.github.com.
• No custom image. The init container runs npm install --prefix /opt/claude @anthropic-ai/claude-code@2.1.316 (8 s, 314M).
• Plugins are opt-in per request ("plugins": ["caveman"]), from an allowlist, each downloaded at a pinned commit and loaded with --plugin-dir. Today: caveman (terse output), ponytail (minimal code) and humanizer (prose without AI tells).

From the n8n pod, the prompt Reply with exactly: pong:

A haiku “pong” logged total_cost_usd 0.00059: that is the API price Claude Code reports, not what I pay. On the subscription it only uses up a little of the plan’s limit.

Two plugin gotchas. Caveman stays in its manual mode under a headless claude -p (the hook sees an SDK entrypoint and opts out), so the runner sets CAVEMAN_DEFAULT_MODE=caveman. And --tools "" also disables the Skill tool, so plugins made only of skills never loaded. The runner now adds Skill whenever plugins are requested.

Building workflows through the API

I do not click workflows together. n8n has a public REST API (/api/v1/workflows), and I build each workflow as JSON, create it inactive, run a test copy, read the execution, then activate it. From my laptop that goes through kubectl exec into the n8n pod, calling localhost:5678, so the API key (read from OpenBao, piped through stdin, never printed) never crosses the login proxy. Every workflow is exported to git, with the Telegram chat id replaced by a placeholder.

Issue 1: First Boot Killed by Its Own Probes

Symptom

Startup probe failed: Get "http://10.0.1.218:5678/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The first container logged Received SIGTERM. Shutting down... in the middle of starting, and kubectl top showed 492m CPU against a 500m limit.

Cause

The first boot runs the database migrations, builds the workflow index and starts the JavaScript task runner, all at once. Throttled at the CPU limit, /healthz answered slower than the chart’s default 1 s probe timeout, and the probes restarted it. A side effect in the log: Task runner connection attempt failed: invalid or expired grant token, because the runner startup outlived its 30 s grant.

The second start finished, since the migrations were done by then.

Fix

CPU limit 1, timeoutSeconds: 5 on all three probes, memory 448Mi request and 896Mi limit. Zero restarts since. Python 3 is missing is only a warning: JavaScript Code nodes work on the internal runner, so I added no task-runner sidecar.

Issue 2: Every External Secret Frozen by a Flux Policy

Symptom

After I added CREDENTIALS_OVERWRITE_DATA to the n8n-app ExternalSecret:

Error: couldn't find key CREDENTIALS_OVERWRITE_DATA in Secret n8n/n8n-app

and on the ExternalSecret:

admission webhook "vpol.validate.kyverno.svc-ignore-..." denied the request: Policy protect-flux-managed failed: Flux-managed object changed outside git by system:serviceaccount:external-secrets:external-secrets

Cause

Without spec.target.template, External Secrets copies the ExternalSecret’s labels onto the Secret it creates, including Flux’s kustomize.toolkit.fluxcd.io/name. My Kyverno policy protect-flux-managed lets only the Flux controllers (and break-glass admins) change objects with that label. So ESO could create the Secret, but its first update was denied.

This was not an n8n problem. Any value rotated in OpenBao would have hit it, and the Secret would have kept the old value without a sound. None of the 12 affected Secrets had been rotated since the policy went to Deny, so nothing had failed yet.

Fix

• Every ExternalSecret got a target.template. An empty engineVersion: v2 is enough: keys and type stay the same, and the labels are no longer copied.
• Each existing labelled Secret was deleted once as a cluster admin, then force-synced. ESO recreated it without the Flux labels.
• Check: a SHA-256 of every Secret’s data before and after, 12 of 12 identical. Labelled ESO Secrets after: 0.

Monitoring

Logs already reached VictoriaLogs through the OTel agent. Metrics needed work: n8n’s /metrics exports only Node.js and process metrics by default, with no execution counts. The N8N_METRICS_INCLUDE_* flags add n8n_workflow_{started,success,failed}_total{workflow_name}, execution durations, webhook and database pool metrics. I read the metric names in n8n’s dist/metrics/prometheus/*.service.js before writing any query, because they are created lazily on the first event and a dashboard on a name that never appears just shows “No data”.

claude-runner logs one JSON line per request (model, status, duration, turns, reported cost, never the prompt), and the runner panels and alerts are VictoriaLogs queries on that line.

• Dashboard Homelab / n8n, 29 panels.
• Alert group homelab-n8n, 6 rules: n8n down, oauth2-proxy down, any workflow failed in 15 min (per workflow), event loop lag p99 over 1 s for 10 min, runner 502s, runner 429s (more than 5 in 15 min).

A test workflow that throws went to firing within about a minute. Two n8n API quirks I hit while testing: a webhook answers 404 for a moment right after activation, and deleting a workflow through the API answers 409 until it is archived.

What It Runs Today

The briefing uses the cheap model because it only rephrases facts that the workflow already collected. The release watch needs judgement (is this changelog line a breaking change for how I use it?), so it goes to Sonnet. Both fall back to sending the raw facts if the model step fails, and a failed GitHub call fails the run so the “workflow failed” alert fires instead of a silent empty message.

The last two rows are the reason I built all of this. They get their own post: an n8n agent pipeline for dvarapala.

Lessons

1. Put the login in front when the app has none. oauth2-proxy plus the identity provider the lab already trusts took less time than evaluating a licence, and a network policy makes sure it cannot be skipped.
2. A subscription token is enough for headless Claude. claude setup-token and claude -p give a frontier model to automations without an API key, at the cost of sharing the plan’s limits with my own use.
3. Keep the shell away from the token. The runners refuse Bash, because the token sits in the process environment. Tools that need a shell get a separate tier with its own, narrower credentials.
4. First boot is the heaviest boot. Size probes and CPU for migrations, not for the steady state, or the probes kill the one start that has to finish.
5. A guardrail can break things quietly. protect-flux-managed did its job, and the job included blocking every rotation. Write a test that exercises the path the policy might block, not only the one it was written for.
6. Read the metric names in the source before writing queries. Lazily created metrics do not exist until the first event.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
40. Part 40: Taking Rate Limiting Back from Cloudflare
41. Part 41: n8n with Claude Code Runners on the Subscription (this post)

References

• n8n: Docs
• n8n: Public REST API
• n8n: Enable Prometheus metrics
• oauth2-proxy: Configuration overview
• Claude Code: Headless mode
• Claude Code: CLI reference
• External Secrets: Templating
• Kyverno: Validate rules
• GitHub: wheresryan22/anatomy