Homelab Rebuild Part 29: One Meter for Every Model, LiteLLM on Workers AI

Intro

I want homelab services to call language models, and I want to know what each call costs before the bill tells me. So the goal was one gateway in the cluster: services get an OpenAI-compatible endpoint and a key, the gateway holds the only provider credential, and every request is counted in dollars and tokens. My laptop stays on its own direct subscriptions; this is for things running in the cluster.

I picked LiteLLM as that gateway and Cloudflare Workers AI as the only provider for now, with as little configuration on the Cloudflare side as possible, so I can move models elsewhere later without touching any client.

TL;DR: LiteLLM v1.105.1 runs in the cluster with 8 Workers AI models and a smart model that lets a small classifier (@cf/cloudflare/clef) pick a tier per request. Every model carries its own per-token price, because LiteLLM’s cost map doesn’t know Workers AI IDs and would report $0. Two things broke on the way: the schema migration Job was OOMKilled at 1 GiB without logging a single error, and the Tailscale proxy for the UI crashlooped with exec format error because one node had a corrupt image layer that deleting the image never removed.

The Figure

The figure is a letter scale and franking machine, because that is what the gateway does to every request: weigh it, send it to the right tray, and stamp its cost on it. Letters wait on the feed table at the left. The scale is the clef classifier: each letter is lifted onto the pan, and the needle swings into the zone marked with one, two or three dots, for simple, medium or complex. The letter then runs along the bed of the franking machine, the die stamps it, and the seven register drums on the front add its price plus the classifier’s to the month’s spend. It ends up in the tray at the front with the same number of dots. Press S, M or C to post a simple, medium or complex letter, click the feed table to post the next one, and press R to reset the register. Drawn with the anatomy skill (GitHub: wheresryan22/anatomy).

The Situation

• k0s 1.36.4 on three Proxmox hosts, Flux owns every app, secrets come from OpenBao through External Secrets.
• CloudNativePG cluster pg with TLS from the lab root CA.
• OTel collectors in front of VictoriaMetrics and VictoriaLogs (Part 28 ).
• LiteLLM v1.105.1, still the Python (FastAPI) proxy. A Rust rewrite is under way upstream, with the full server targeted for December 2026; it reads the same YAML config, so mine should carry over.

flowchart LR
  S[Homelab service<br/>virtual key] --> L[LiteLLM<br/>ns litellm]
  L -->|smart| C[clef classifier]
  C --> L
  L -->|one token| CF[Workers AI<br/>OpenAI-compatible API]
  L --> PG[(Postgres<br/>keys, spend)]
  L --> OT[OTel collector] --> VM[(VictoriaMetrics)]

Setup

Why Workers AI only

Cloudflare offers two ways in. Workers AI runs models on Cloudflare’s own GPUs (the @cf/... IDs) and bills them as Workers usage. AI Gateway Unified Billing proxies third-party models (OpenAI, Anthropic, Google, xAI) at list price with no token markup, but takes a 5% fee when you buy credits. I don’t need top-tier models for homelab automation, so I started with Cloudflare’s own catalog and skipped Unified Billing.

The Cloudflare side is one API token with Account > Workers AI > Read. That is all. Every model uses the same OpenAI-compatible endpoint:

https://api.cloudflare.com/client/v4/accounts/<account-id>/ai/v1

@cf/... models also need a cf-aig-gateway-id header; default works.

The models

Clients call a model by its short name. Moving glm-5.3 to another provider later means changing its model, api_base and api_key; the name and every client key stay the same.

One YAML anchor per provider

All Workers AI entries share one block through a YAML anchor:

model_list:
  - model_name: glm-4.7-flash
    litellm_params: &cf
      model: openai/@cf/zai-org/glm-4.7-flash
      api_base: os.environ/CF_AI_BASE
      api_key: os.environ/CLOUDFLARE_API_KEY
      extra_headers:
        cf-aig-gateway-id: default
    model_info: {input_cost_per_token: 6.0e-8, output_cost_per_token: 4.0e-7}
  - model_name: glm-5.3
    litellm_params: {<<: *cf, model: openai/@cf/zai-org/glm-5.3}
    model_info: {input_cost_per_token: 1.4e-6, cache_read_input_token_cost: 2.6e-7, output_cost_per_token: 4.4e-6}

The openai/ prefix tells LiteLLM to speak the OpenAI protocol to whatever api_base says. The model_info prices matter: LiteLLM looks costs up in its built-in map by model ID, and openai/@cf/zai-org/glm-5.3 isn’t in it. Without model_info, every call would be logged as $0, and a gateway whose job is counting money would count nothing.

The smart model

smart is LiteLLM’s auto_router/complexity_router. With the JEV classifier pointed at a model in my own list, it asks clef how hard the prompt is and sends it to one model from that tier:

- model_name: smart
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      classifier_type: jev
      jev_classifier_config:
        deployment_name: clef
      tiers:
        SIMPLE: [glm-4.7-flash]
        MEDIUM: [glm-5.3-flash, deepseek-v4-pro, qwq-32b]
        COMPLEX: [kimi-k2.6, kimi-k2.7-code, glm-5.3]
        REASONING: [kimi-k2.6, kimi-k2.7-code, glm-5.3]

If clef fails or takes longer than 3 seconds, a circuit breaker opens for 30 seconds and the router falls back to a keyword heuristic, so a classifier outage slows nothing down.

Shipping it with Flux

• Chart: litellm-helm 1.1.3 isn’t published to any registry I could find, so Flux reads it straight from the LiteLLM repository with a GitRepository at tag v1.105.1 and a sparse checkout of helm/litellm-helm. The image is pinned by digest and verified with the repository’s cosign.pub.
• Database: a litellm database and role on CNPG. Prisma connects with sslmode=require&sslcert=/etc/lab-ca/ca.crt&sslaccept=strict, so it checks the server certificate against the lab CA. I wasn’t sure Prisma would accept that; it did on the first try.
• Secrets: the Cloudflare token, master key and salt key come from OpenBao. The ExternalSecret template runs trim on the token, because the first copy I stored had a trailing newline from stdin.
• Egress: a CiliumNetworkPolicy allows only api.cloudflare.com:443, the Postgres pods, the OTel collector and DNS.
• Access: the UI at https://litellm.franzramadhan.lab through lab-gateway, and on the tailnet through a Tailscale Ingress. The API is in-cluster only.
• Budget: max_budget: 50 per 30d for the whole proxy. Each service will get a virtual key with its own smaller budget.
• Metrics: the otel callback with LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true. LiteLLM sends spans too, so the collector got a traces pipeline that exports to nop: accept and drop, instead of rejecting every export.

Issue 1: The Migration Job Died Without a Word

Symptom

Flux reported the install as failed before the proxy ever started:

Helm install failed for release litellm/litellm with chart litellm-helm@1.1.3:
failed pre-install: failed early due to stalled resources:
[Job/litellm/litellm-migrations status: 'Failed']

By then the Job and its pods were gone, so kubectl logs had nothing. The logs were still in VictoriaLogs, and they looked like success. Every attempt applied or confirmed the migrations and then stopped:

# k8s.container.name:prisma-migrations
195 migrations found in prisma/migrations
No pending migrations to apply.
litellm_proxy_extras - INFO - Request-log indexes are all in place

No error, no traceback, and four attempts with the same ending.

Dead end: the TLS connection string

That sslaccept=strict URL was the change I trusted least, so it was my first suspect. The logs ruled it out: Prisma printed Datasource "client": PostgreSQL database "litellm" ... at "pg-rw.databases.svc:5432" and applied 195 migrations on the first attempt. The database side was fine.

Cause

A container that stops mid-run without logging anything is usually killed from outside. The exit reason only exists while the pod does, so I forced a retry and watched the pod’s status:

kubectl -n litellm get pods -l job-name=litellm-migrations -o json \
  | jq '.items[].status.containerStatuses[].lastState.terminated'
# {"exitCode": 137, "reason": "OOMKilled", ...}

The chart’s migration Job runs litellm/proxy/prisma_migration.py, and that script starts with run_server(("--skip_server_startup",), ...): it loads the entire proxy, config and all, just to run the migrations and exit. With my 1 GiB limit, it finished the migrations and then got killed while the rest of the proxy was still loading.

Fix

2 GiB limit for the migration Job and the proxy (1 GiB request for the proxy):

migrationJob:
  resources:
    requests: {cpu: 100m, memory: 512Mi}
    limits: {cpu: "1", memory: 2Gi}

The next upgrade went through.

Issue 2: The Corrupt Layer That Wouldn’t Leave

Symptom

The Tailscale Ingress for the UI never got a hostname. Its proxy pod was in CrashLoopBackOff on k0s-pve-01-worker-1:

# kubectl -n tailscale logs ts-litellm-...-0 --previous
exec /usr/local/bin/containerboot: exec format error

Same image (tailscale/tailscale:v1.102.4), same node, same error as the tailnet DNS proxy in Part 28 , one day earlier.

Dead end: deleting the image

In Part 28 I deleted the cached image and the pod recovered. This time I did the same and the pod still failed. (k0s 1.36 has no crictl subcommand; the working command is k0s ctr -n k8s.io.)

k0s ctr -n k8s.io images rm docker.io/tailscale/tailscale:v1.102.4
kubectl -n tailscale delete pod ts-litellm-...-0
# exec /usr/local/bin/containerboot: exec format error

So I looked at the binary itself, inside containerd’s overlayfs snapshots:

find /opt/lib/k0s/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots \
  -path '*/usr/local/bin/containerboot' -exec ls -l {} \;
# -r-xr-xr-x 1 root root 0 Jan  1  1970 .../snapshots/1099/fs/usr/local/bin/containerboot

Zero bytes. The 1970 date is normal for this image (it’s built with fixed timestamps); the size is the problem. The kernel can’t recognize an empty file as anything, hence “exec format error”. The snapshot directory was created on 2026-10-13 at 20:42, the night pve-01 crashed (Part 27 ). The layer was unpacked before the crash, and its file contents never reached the disk.

Cause

Deleting an image removes the image record. The unpacked layer is a separate snapshot, and containerd keeps it while anything still references it. Every re-pull matched the same layer digest, found the snapshot already “unpacked”, and reused the empty files. Even images rm --sync after moving the pod away left snapshot 1099 in place, and a fresh ctr images pull created no new snapshot at all.

That also explains Part 28: the DNS proxy recovered because it was rescheduled onto another worker, not because the image delete fixed anything.

Fix

Find the layer’s snapshot key, then delete the snapshot itself:

k0s ctr -n k8s.io image usage --snapshotter overlayfs docker.io/tailscale/tailscale:v1.102.4
# KEY                                                                     SIZE     INODES
# sha256:853c7f02ac0ee4720c379dc50ce7b250bae88c8a77d660c57ecfbafa5b1c1c8f 97.6 MiB 7
# ...
k0s ctr -n k8s.io images rm docker.io/tailscale/tailscale:v1.102.4
k0s ctr -n k8s.io snapshots rm sha256:853c7f02ac0ee4720c379dc50ce7b250bae88c8a77d660c57ecfbafa5b1c1c8f
k0s ctr -n k8s.io images pull docker.io/tailscale/tailscale:v1.102.4
# -r-xr-xr-x 1 root root 26249110 Jan  1  1970 .../snapshots/1870/fs/usr/local/bin/containerboot

The 7-inode layer is the one holding the Tailscale binaries. After the delete and a fresh pull, a new snapshot (1870) had a 26 MB containerboot, and 1099 was gone. I scanned every other snapshot on that worker for empty executables and found none. The UI answered on the tailnet with HTTP 200.

Results

Each model got the same prompt (“Reply with the single word pong”). Times are LiteLLM’s own request durations from its spend log, in whole seconds:

• smart sent the trivial prompt to the SIMPLE tier, as it should. The classifier call (253 tokens, $0.000061) cost more than twice the answer ($0.000027). For short prompts, smart is the expensive way to ask.
• qwq-32b puts its reasoning into content; the others return it separately in reasoning_content.
• With max_tokens: 300, smart once returned content: null: the reasoning used up the whole budget before any answer. Reasoning models need room.
• Total spend for all testing: $0.0034 of the $50 budget.
• VictoriaMetrics receives gen_ai_client_operation_duration_seconds, gen_ai_client_token_usage and gen_ai_usage_cost_USD, so cost per model can go on a dashboard next to everything else.
• The edge probes from Part 28 now include https://litellm.franzramadhan.lab/health/liveliness.

Lessons

1. A gateway that counts money needs prices for every model. LiteLLM only knows costs for model IDs in its built-in map. Proxied or renamed models need model_info, or spend reads $0.
2. A container that dies without logging was probably killed. Check lastState.terminated while the pod still exists; once a Job is deleted, the exit reason goes with it.
3. Read what the migration Job actually runs. This one boots the whole proxy, so it needs the proxy’s memory, not a script’s.
4. Deleting an image doesn’t delete its layers. If a binary is broken on one node, check its size in the snapshot, then remove the snapshot by key.
5. A fix that works because the pod moved isn’t a fix. Part 28’s “fix” hid the corrupt layer for a day.
6. A classifier isn’t free. For short prompts, calling the cheapest model directly costs less than letting smart decide.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI (this post)

References

• GitHub: wheresryan22/anatomy
• LiteLLM: Proxy configuration
• LiteLLM: Custom pricing
• LiteLLM: Helm chart
• GitHub: BerriAI/litellm#31263, Rust migration
• Cloudflare: Workers AI models
• Cloudflare: Workers AI OpenAI-compatible endpoints
• Flux: GitRepository sparse checkout
• containerd: ctr snapshots
• Kubernetes: Debug running pods