Intro
I wanted one place to run small automations against the lab. I had three in mind: a morning summary of what broke overnight, a watch on upstream releases of what I run, and later, agents that work on my own code. Cron jobs and shell scripts can do the first two. But every new one means another image, another CronJob and another set of secrets. A workflow engine with a UI, an API and retries is the better shape.
n8n is that engine. This post covers how it sits in the cluster and how its steps call language models. Cheap ones go through LiteLLM, and Claude goes through headless Claude Code on my subscription. It also covers a LinkedIn inbox triage that drafts replies for my approval, the errors on the way, and what n8n runs today.
TL;DR: n8n 2.45.7 runs in namespace
n8n, with its database on the CloudNativePG cluster. It is reachable on the tailnet and the LAN only. It sits behind oauth2-proxy with Cloudflare Access and Google, because n8n Community has no SSO. Simple steps call LiteLLM with a virtual key that n8n pre-fills into every OpenAI credential. Hard steps callclaude-runner, a 170-line Node wrapper aroundclaude -p, with one Deployment per model tier. It authenticates with aclaude setup-tokensubscription token, never an API key. The first boot died to its own liveness probe. Also, a Kyverno policy I wrote two weeks ago silently blocked every External Secret update. LinkedIn messages arrive as Gmail notification mails. Workers AI’sclefdecision model labels them. A job offer gets a drafted decline that waits for my approval on Telegram. The workflow sends nothing to LinkedIn.
The Figure
I drew n8n as a Jacquard loom. The chain of six punched cards on the table at the right is one workflow, a card per node. The clock on the head turns to the trigger hour to start it. The card under the reader lights and lifts its hook. A cheap step spins the side bobbin on the left (LiteLLM on Workers AI). A hard step throws a shuttle from one of the three boxes on the right. Haiku has three small shuttles, sonnet two and opus one big one. When every shuttle in a box is already out, its red flag goes up (a 429). The card then waits a beat before n8n tries again. The cloth that rolls onto the front beam and the three lamps (Telegram, email and GitHub, left to right) show what n8n delivered. Click the figure, or press 1, 2 or 3, to load the morning briefing, the release watch or the dvarapala review. Built with the anatomy skill.
The Situation
- k0s cluster on three Proxmox hosts, FluxCD owns every app (Part 16).
- CloudNativePG cluster
pg, two instances on NVMe data disks (Part 8). - Secrets in OpenBao, synced by External Secrets (Part 5).
- Cloudflare Access with Google as the OIDC issuer for OpenBao, Headlamp, LiteLLM and the Proxmox UI (Part 19).
- LiteLLM in front of Cloudflare Workers AI (Part 29).
- Kyverno policy
protect-flux-managedin Deny mode since Part 34.
| Piece | Version |
|---|---|
| n8n | 2.45.7, chart 8gears/n8n 2.1.1 (OCI), image pinned by digest |
| oauth2-proxy | 7.15.5, chart 10.7.1 |
| Claude Code (runners) | 2.1.316, installed by an init container on node:24-bookworm-slim |
| Database | n8n on CNPG pg (PostgreSQL 18), TLS verified against the lab root CA |
Setup
A login n8n does not have
n8n Community has no OIDC or SAML login: both need a paid licence. So nobody reaches n8n directly. oauth2-proxy sits in front of it, with the same Cloudflare Access plus Google pattern as everything else in the lab.
flowchart LR
Me["Mac / phone"] -->|tailnet| TS["Tailscale Ingress"]
Me -->|home LAN| GW["lab-gateway<br/>n8n.franzramadhan.lab"]
TS --> OP["oauth2-proxy"]
GW --> OP
OP -->|OIDC| CFA["Cloudflare Access<br/>(Google)"]
OP --> N8N["n8n"]
N8N --> PG[("CNPG pg")]- Tailnet: a Tailscale
Ingressto the oauth2-proxy Service. LAN: anHTTPRouteonlab-gatewaywith a lab CA certificate. Nothing on the internet. - The Access policy allows one email. oauth2-proxy checks it again with an
emails file and
emailDomains: []. The chart default is["*"]. That lets any Google account through after Access lets it in. - No fixed
redirect-url. Withreverse-proxy: true, the callback follows the host that the login started on, so the cookie lands on the right one. Both callback URLs are registered in the Access app. - A CiliumNetworkPolicy lets only oauth2-proxy (and
monitoring) reach n8n. So nothing inside the cluster can skip the login. /webhook/*skips oauth2-proxy (skip-auth-route), so webhook triggers still work. They are only reachable from the tailnet and the LAN.
The result is two logins in a row: Google, then n8n’s own owner account.
n8n’s session cookie lasts 7 days. I kept it that way on purpose. The
alternative is an EXTERNAL_HOOK_FILES hook that trusts X-Forwarded-Email.
That hook uses internal APIs and breaks on upgrades.
LiteLLM for the cheap steps
n8n gets its own LiteLLM virtual key, limited to smart, glm-4.7-flash,
glm-5.3-flash and clef, with a budget of 5 USD per 30 days. I do not type
it into the UI. Instead, CREDENTIALS_OVERWRITE_DATA (templated by the
ExternalSecret) pre-fills every n8n OpenAI credential:
{"openAiApi": {"apiKey": "<from OpenBao>", "url": "http://litellm.litellm.svc:4000/v1"}}
A new OpenAI credential in n8n now needs no typing. The key never leaves OpenBao and the Secret.
Two things bit me in the first workflow. First, smart can route to a
reasoning model. With max_tokens 400, the reasoning used the whole budget,
and the answer came back empty (now 2000). Second, LiteLLM caches identical
prompts. So a summary prompt that did not change between runs returned
yesterday’s answer. The prompt now includes the current time.
Claude on the subscription, not an API key
For steps that need a frontier model, I wanted Claude. I wanted it billed to
the Claude subscription that I already pay for, not per token. Claude Code
can run headless with a subscription token. claude setup-token prints a
long-lived OAuth token, and the CLI reads it from CLAUDE_CODE_OAUTH_TOKEN.
Runs count against the plan’s usage limits, shared with my interactive use.
--bare is not an option here. In bare mode, authentication is strictly
ANTHROPIC_API_KEY or apiKeyHelper.
claude-runner/server.mjs is a small HTTP wrapper with only Node’s standard
library:
POST /run {prompt, system_prompt?, max_turns?, tools?, plugins?}
-> claude -p --model <tier> --output-format json --max-turns N
--no-session-persistence --strict-mcp-config --tools <list>
-> Claude's JSON result (result, num_turns, total_cost_usd, ...)
The prompt goes in on stdin, and every request gets a fresh scratch directory. Each tier is its own Deployment and Service:
| Tier | Model | Parallel runs | Timeout | Memory limit |
|---|---|---|---|---|
| haiku | claude-haiku-5-5 | 3 | 5 min | 1536Mi |
| sonnet | claude-sonnet-5-5 | 2 | 10 min | 1Gi |
| opus | claude-opus-5-5 | 1 | 30 min | 1Gi |
- A busy tier answers 429 instead of queueing. n8n’s Retry On Fail handles it.
- Tools default to none. Allowed:
Read,Write,Edit,Glob,Grep,WebSearch. The runner refusesBash. The token is in the process environment, and a shell can print it. - Read-only root filesystem, non-root, all capabilities dropped,
automountServiceAccountToken: false, a scratchemptyDirwith a size limit. - The CiliumNetworkPolicy lets only n8n call the runners. The runners reach
api.anthropic.com, and the init containerregistry.npmjs.organdcodeload.github.com. - No custom image. The init container runs
npm install --prefix /opt/claude @anthropic-ai/claude-code@2.1.316(8 s, 314M). - Plugins are opt-in per request (
"plugins": ["caveman"]), from an allowlist. The runner downloads each one at a pinned commit and loads it with--plugin-dir. Today: caveman (terse output), ponytail (minimal code) and humanizer (prose without AI tells).
From the n8n pod, the prompt Reply with exactly: pong:
| Tier | Result | Duration |
|---|---|---|
| haiku | pong | 2988 ms |
| opus | pong | 4499 ms |
A haiku “pong” logged total_cost_usd 0.00059. That is the API price that
Claude Code reports, not what I pay. On the subscription, it only uses up a
little of the plan’s limit.
Two plugin gotchas. Caveman stays in its manual mode under a headless
claude -p (the hook sees an SDK entrypoint and opts out). So the runner sets
CAVEMAN_DEFAULT_MODE=caveman. Also, --tools "" disables the Skill tool,
so plugins made only of skills never loaded. The runner now adds Skill when
a request asks for plugins.
Building workflows through the API
I do not click workflows together. n8n has a public REST API
(/api/v1/workflows). I build each workflow as JSON and create it inactive.
Then I run a test copy, read the execution, and activate it. From my laptop,
that goes through kubectl exec into the n8n pod, which calls
localhost:5678. So the API key (read from OpenBao, piped through stdin,
never printed) never crosses the login proxy. I export every workflow to git,
with a placeholder for the Telegram chat id.
LinkedIn messages without a LinkedIn API
I wanted my LinkedIn inbox sorted into four groups: spam, job offers, casual messages (friends and first connections) and the rest. Job offers should get a polite decline, ready to send. But LinkedIn gives a member no way to read their own messages. Member OAuth can post, but it cannot read DMs. The Community Management API is for company pages. The Messaging API is for partners only. Unofficial cookie based APIs break LinkedIn’s terms, so I did not use them.
LinkedIn does mail a notification for every new message, with the text and a link to the thread. Those mails are the input.
flowchart LR
T["every 3 h<br/>09:00 to 21:00"] --> GL["Gmail list<br/>from:linkedin.com newer_than:2d"]
GL --> D["dedup<br/>(static data, 7 days)"]
D --> P["parse<br/>sender, text, thread link"]
P --> C["clef<br/>/v1/decisions"]
C --> L{"label"}
L -->|unsure| F["glm-4.7-flash"]
F --> L
L -->|spam| X["dropped"]
L -->|job_offer| A["Telegram: approve<br/>drafted decline"]
A -->|Use reply| R["Telegram: reply text<br/>+ thread link"]
L -->|casual, other| N["Telegram notice"]- Gmail, read only. n8n’s Gmail credential asks for the modify and send
scopes. So I used the generic Google OAuth2 API credential with
gmail.readonlyonly. Its allowed HTTP request domains are set togmail.googleapis.com. The OAuth app is in production but unverified. Verification of a restricted scope needs a security assessment. An unverified app works for up to 100 users, behind a “Google hasn’t verified this app” screen. A production app also keeps its refresh token. A testing app loses it after 7 days. - Classifier:
clefon Workers AI through LiteLLM.clefis a decision model, not a chat model. It answers yes/no questions with a probability. LiteLLM serves it on/v1/decisions. - Job offer is checked before spam. A real offer scores high on spam
too. If spam came first, real offers would be dropped. The order is:
job_offerat 0.7 or more, thenspamat 0.8 or more, thencasualat 0.5 or more, elseother. Unsure scores go toglm-4.7-flashas a chat prompt: ajob_offerbetween 0.3 and 0.7, aspambetween 0.5 and 0.8, or a failed call. It getsmax_tokens1000. At 400, its reasoning used the whole budget and the answer came back empty. - Nothing is sent to LinkedIn. A job offer gets a fixed decline with the sender’s first name. It goes to Telegram with n8n’s send-and-wait and double approval (“Use reply” or “Skip”, 3 days to answer). After “Use reply”, the workflow sends the reply text and the thread link. I paste the reply in LinkedIn myself. Casual and other messages get a one-line Telegram notice. Spam is dropped.
The clef request, three questions in one call:
{"model": "clef", "input": "From: <name> (InMail, not a connection)\n\n<message>",
"questions": [
{"type": "predicate", "name": "job_offer", "instructions": "<yes/no question>"},
{"type": "predicate", "name": "spam", "instructions": "<yes/no question>"},
{"type": "predicate", "name": "casual", "instructions": "<yes/no question>"}]}
Scores on test messages through n8n’s key:
| Sample | job_offer | spam | casual |
|---|---|---|---|
| explicit job offer (InMail) | 0.98 | 0.61 | 0.02 |
| vague recruiter (“open to something new?”) | 0.94 | 0.44 | 0.09 |
| friend asking for coffee | 0.01 | 0.03 | 0.95 |
| growth hacking spam (InMail) | 0.02 | 0.99 | 0.02 |
| endorsement request | 0.01 | 0.03 | 0.70 |
The first live message was a recruiter InMail for a contract DevOps role. It
scored job_offer 0.98, spam 0.51 and casual 0.02. It reached Telegram
as a drafted decline. I tapped “Use reply” twice, and the reply text and
thread link followed.
Issue 1: First Boot Killed by Its Own Probes
Symptom
Startup probe failed: Get "http://10.0.1.218:5678/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
The first container logged Received SIGTERM. Shutting down... in the middle
of starting, and kubectl top showed 492m CPU against a 500m limit.
Cause
The first boot runs the database migrations, builds the workflow index and
starts the JavaScript task runner, all at once. The pod was throttled at the
CPU limit. So /healthz answered slower than the chart’s default 1 s probe
timeout, and the probes restarted it. A side effect in the log: Task runner connection attempt failed: invalid or expired grant token. The runner
startup took longer than its 30 s grant.
The second start finished, because the migrations were done by then.
Fix
I set the CPU limit to 1, timeoutSeconds: 5 on all three probes, and memory
to a 448Mi request and an 896Mi limit. There were zero restarts after that.
Python 3 is missing is only a warning. JavaScript Code nodes work on the
internal runner, so I added no task-runner sidecar.
Issue 2: Every External Secret Frozen by a Flux Policy
Symptom
After I added CREDENTIALS_OVERWRITE_DATA to the n8n-app ExternalSecret:
Error: couldn't find key CREDENTIALS_OVERWRITE_DATA in Secret n8n/n8n-app
and on the ExternalSecret:
admission webhook "vpol.validate.kyverno.svc-ignore-..." denied the request: Policy protect-flux-managed failed: Flux-managed object changed outside git by system:serviceaccount:external-secrets:external-secrets
Cause
Without spec.target.template, External Secrets copies the ExternalSecret’s
labels onto the Secret it creates. That includes Flux’s
kustomize.toolkit.fluxcd.io/name. My Kyverno policy protect-flux-managed
lets only the Flux controllers (and break-glass admins) change objects with
that label. So ESO (External Secrets Operator) was able to create the Secret, but
Kyverno denied its first update.
This was not an n8n problem. Any value rotated in OpenBao hits it, and the Secret then keeps the old value without a sound. Nobody rotated any of the 12 affected Secrets after the policy went to Deny, so nothing failed before.
Fix
- Every ExternalSecret got a
target.template. An emptyengineVersion: v2is enough. Keys and type stay the same, and ESO no longer copies the labels. - I deleted each existing labelled Secret once as a cluster admin, then force-synced it. ESO recreated it without the Flux labels.
- Check: a SHA-256 of every Secret’s
databefore and after, 12 of 12 identical. Labelled ESO Secrets after: 0.
Issue 3: clef Is Not a Chat Model
Symptom
Calling clef through chat completions, like every other LiteLLM step:
AiError: Bad input: Error: required properties at '/' are 'model,state,questions'
(Workers AI code 5006, passed through by LiteLLM as a 400.)
Cause
clef takes a state and a list of questions, not messages. LiteLLM’s own
complexity router calls it with a state and a question map, and that shape
on /v1/decisions fails too:
Invalid Decisions request: 2 validation errors for OpenAIDecisionRequestBody
input
Field required
questions
Input should be an instance of Sequence
/v1/decisions takes the OpenAI shape (input plus a questions list of
predicates). The state plus map form is the router’s internal one.
Fix
The request shown in Setup. The n8n key also needed clef in its model list
(/key/update with the master key; /key/info is a GET with ?key=, a POST
answers 405).
Issue 4: The Classifier Saw Only the Subject
Symptom
On the first real InMail, the parsed text was the subject and the sender’s
name. The message itself was missing, and the thread link had & in its
query string.
Cause
The text part of a LinkedIn notification is: the subject twice, the sender’s
name, “Reply”, the thread link, the message, a dashed line, the footer. A cut
at the first “Reply” lands before the message. The link came from the HTML
part, still entity-encoded. The sender name from the From header also ended
in " via LinkedIn".
Fix
Take the text after the thread link and stop at a dashed line (or the footer
phrases), at most 2,000 characters. Decode & in the link and strip
" via LinkedIn" from the name.
Issue 5: A LinkedIn Link Telegram Could Not Parse
Symptom
The first live job offer went through the approval, the reply text arrived, and the thread link node failed:
Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 403
Cause
n8n’s Telegram node sends with parse_mode Markdown by default. LinkedIn’s
notification links carry tracking parameters with underscores, and Markdown
read the first one as the start of italic text that never ended.
Fix
Every Telegram node in the workflow sends HTML, with &, < and > escaped
in the text. A test copy sent a link with underscores, & and <test>, and
it arrived unchanged.
Monitoring
Logs already reached VictoriaLogs through the OTel agent. Metrics needed
work. By default, n8n’s /metrics exports only Node.js and process metrics,
with no execution counts. The N8N_METRICS_INCLUDE_* flags add
n8n_workflow_{started,success,failed}_total{workflow_name}, execution
durations, webhook and database pool metrics. Before I wrote any query, I
read the metric names in n8n’s dist/metrics/prometheus/*.service.js. n8n
creates them lazily on the first event. A dashboard on a name that never
appears only shows “No data”.
claude-runner logs one JSON line per request (model, status, duration,
turns, reported cost, never the prompt). The runner panels and alerts are
VictoriaLogs queries on that line.
- Dashboard Homelab / n8n, 29 panels.
- Alert group
homelab-n8n, 6 rules: n8n down, oauth2-proxy down, any workflow failed in 15 min (per workflow), event loop lag p99 over 1 s for 10 min, runner 502s, runner 429s (more than 5 in 15 min).
A test workflow that throws went to firing within about a minute. I hit two n8n API quirks in the tests. A webhook answers 404 for a moment right after activation. A delete of a workflow through the API answers 409 until you archive the workflow.
What It Runs Today
| Workflow | When | Steps | Output |
|---|---|---|---|
| Morning lab briefing | 07:00 daily | Grafana API: firing alerts, alert state changes of the last 12 h, VictoriaMetrics queries (pod restarts, age of the last Postgres backup) -> LiteLLM smart writes at most 6 lines | Telegram and email |
| Upstream release watch | 07:00 daily | GitHub releases and advisories of 11 projects I run, last 24 h -> claude-runner-sonnet marks each one SECURITY, BREAKING or routine | Telegram and email, nothing when nothing is new |
| dvarapala review | 08:00 Mon, Wed, Fri | claude-runner-opus reviews my open source Rust project | GitHub, Telegram |
| dvarapala implement | 07:30 and 13:30 | a dev runner with git and gh turns approved items into pull requests | GitHub, Telegram |
| LinkedIn inbox triage | every 3 h, 09:00 to 21:00 | Gmail notification mails -> clef labels (fallback glm-4.7-flash) -> drafted decline for job offers | Telegram, approval before any reply |
The briefing uses the cheap model because it only rephrases facts that the workflow already collected. The release watch needs judgement (is this changelog line a breaking change for how I use it?), so it goes to Sonnet. If the model step fails, both send the raw facts instead. A failed GitHub call fails the run. So the “workflow failed” alert fires instead of a silent empty message.
The last two rows are the reason I built all of this. They get their own post: an n8n agent pipeline for dvarapala.
Lessons
- Put the login in front when the app has none. oauth2-proxy plus the identity provider that the lab already trusts took less time than an evaluation of a licence. A network policy makes sure that nobody can skip it.
- A subscription token is enough for headless Claude.
claude setup-tokenandclaude -pgive a frontier model to automations without an API key. The cost is that automations share the plan’s limits with my own use. - Keep the shell away from the token. The runners refuse
Bash, because the token sits in the process environment. Tools that need a shell get a separate tier with its own, narrower credentials. - First boot is the heaviest boot. Size probes and CPU for migrations, not for the steady state. If you do not, the probes kill the one start that must finish.
- A guardrail can break things quietly.
protect-flux-manageddid its job, and the job included a block on every rotation. Write a test for each path that the policy can block, not only the path you wrote it for. - Read the metric names in the source before you write queries. Lazily created metrics do not exist until the first event.
- Order the questions by what a miss costs. A recruiter’s message looks like spam to a classifier too. Job offers go first, so a real offer is never dropped. An unsure score goes to a second model.
- When there is no API, try the inbox. The notification mails carry everything the triage needs, behind a read-only scope. A manual reply keeps the account within LinkedIn’s terms.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
- Part 40: Taking Rate Limiting Back from Cloudflare
- Part 41: n8n with Claude Code Runners on the Subscription (this post)
References
- n8n: Docs
- n8n: Public REST API
- n8n: Enable Prometheus metrics
- oauth2-proxy: Configuration overview
- Claude Code: Headless mode
- Claude Code: CLI reference
- External Secrets: Templating
- Kyverno: Validate rules
- GitHub: wheresryan22/anatomy
- Google: Gmail API users.messages.list
- Google: Unverified apps
- LinkedIn: Getting access to LinkedIn APIs
- Telegram: Bot API formatting options
