Homie: One Page for My Whole Homelab, and a Public Demo of It

Intro

Answering “is my homelab healthy?” meant four tabs: the Proxmox UI, Headlamp for Kubernetes and Flux, two Grafanas for alerts, and log queries for the security tools. During an outage it got worse, because the UIs that run in the cluster go down with it. And my lab powers off every night from 23:30 to 06:00, so “down” is a normal daily state that every tool shows in red.

So I wrote Homie, an open source dashboard that puts every layer of the lab on one page, with a few safe actions and deep links to the specialist UIs for everything else. Then I opened a read-only demo of it to the internet.

TL;DR: Homie is a single Rust binary with the web UI embedded and no database. It reads Proxmox, any Kubernetes cluster, Flux and Argo CD, Helm, any number of Grafanas, VictoriaLogs or Loki, Kyverno, Tetragon, Wazuh, Tailscale, healthchecks.io and GitHub Actions, all optional. It went from the first RFC to v0.5.0 in one day, one RFC per feature. In my cluster it uses 5 MiB of memory. The public demo at homie-demo.franzramadhan.dev is a separate pod whose only login is a read-only role, enforced twice (Homie’s filter and Kubernetes RBAC), behind the Cloudflare Tunnel I already had and the free plan’s one rate limiting rule.

The Figure

The dollhouse is the lab, and each floor is a layer Homie puts on one page. The cellar holds three boilers, the Proxmox nodes; the small one is pve-03, which only holds quorum. The ground floor holds eight bunks, the k0s VMs; the three four-poster beds are the controllers. The first floor is GitOps: a ledger desk and a screw press, Flux. The attic holds alerts, security and the tailnet: a bell, a lantern and a radio. Click the figure or press Space to switch viewers. The owner sees every floor, with the power switch free. The demo role gets a padlock on the switch and a louvred shutter over the attic. In quiet hours the front closes and the lamps turn grey, not red. The figure was built with the anatomy skill.

The Situation

• Homie v0.5.0, MIT, on GitHub: Rust backend, a Vite, React and TanStack SPA embedded in the binary. The binary is 7.5 MB, the amd64 image about 4.5 MB compressed.
• Lab: three Proxmox VE 9.2 hosts, a k0s cluster on 8 VMs (3 controllers, 5 workers), Flux (Part 16), secrets from OpenBao (Part 5).
• Login: Cloudflare Access with Google as the OIDC provider, the same one Headlamp uses (Part 20).
• Alerts: the in-cluster Grafana and Grafana Cloud (Part 32).
• Public path: the Cloudflare Tunnel to public-gateway (Part 9), unused until now.

RFC First, Even for a One-Day Project

The first commit was not code. It was RFC 0001: the problem, goals, non-goals, every config key, security considerations, alternatives and the exit criteria for v0.1.0. After that, every feature started as its own RFC, merged as accepted before any code, and every deviation found while building went back into it.

16 RFCs so far: 11 implemented, 4 rejected and kept for the record (Flux suspend, configurable security presets, Alertmanager, guest migration). A rejected RFC is cheap and still useful: the next time I want guest migration I can read why I said no.

Every release was tested in the lab first: a manual workflow builds homie:dev-<sha>, I pin its digest in the infrastructure repo, Flux rolls it out, and a local end-to-end script logs in with a password and checks every page and the refusals; with --actions it also power-cycles a sandbox VM through Homie.

How It Works

flowchart LR
  B[Browser: Homie SPA] -- "same origin, cookie" --> H[homie binary]
  H --> C[TTL cache and last-known state]
  H -- "API token" --> P[Proxmox VE]
  H -- "user's ID token" --> K[kube-apiserver]
  H -- "Viewer token" --> G[Grafana, Grafana Cloud]
  H -- "API keys" --> O[Tailscale, healthchecks.io, GitHub]

• A backend is not optional. Neither the Proxmox API nor kube-apiserver sends CORS headers, and credentials must never reach the browser. The browser only talks to Homie.
• Named queries, not a proxy. The API has a fixed list of queries (pve.resources, k8s.flux, security.wazuh, …) and actions (guest power, Flux reconcile). Each returns { data, fetched_at, stale, quiet, error }. That is an allowlist by construction, and backend URLs and query languages stay out of the browser.
• Cached and stale-tolerant. Results are cached for 10 to 30 seconds, so several tabs don’t multiply the load. A failed fetch returns the last good result marked stale. On a laptop, state_dir also keeps a snapshot on disk, so Homie still shows the last known state when the whole lab is off.
• Quiet hours. Inside 23:30-06:00 (Asia/Jakarta), anything marked quiet that is unreachable shows as grey “scheduled off”, not red, and its alerts are listed but not counted. Grafana Cloud and healthchecks.io are not quiet: they should never go dark. Red every night teaches you to ignore red.
• No standing cluster-admin. With forward_token, Homie calls kube-apiserver with the logged-in user’s own ID token, so the audit log says oidc:<email>, not a ServiceAccount. kube-apiserver needed a second jwt issuer in its structured authentication config; it reloads that file on change, so I copied it to the controllers one at a time with no restart.
• Proxmox power, but not config. A homie@pve user with the PVEAuditor privileges plus VM.PowerMgmt: it can start, shut down and stop guests, and cannot change or delete them. Guests tagged homie-protected (my three controllers, since two down loses etcd quorum) need their name typed before any power action.

Issue 1: Every Power Action Failed

Symptom

The first power test against the real cluster failed for every guest, and the failed-task list was empty, though I knew there were failed tasks.

Cause

The helper that joins a base URL and a path percent-encoded the ? in /cluster/resources?type=vm. Proxmox answered 501, the action could not find the guest, and the task query failed the same way, silently.

Fix

Split the query string off before joining, with a regression test. Failed tasks now come from the last 24 hours only, so a week-old failure does not keep the overview amber.

Issue 2: Shut Down Did Not Shut Down

Symptom

I clicked Shut down on a VM. Nothing happened, and a second click failed with can't lock file .../lock-<vmid>.conf.

Cause

Proxmox’s shutdown asks the guest (through the guest agent) to power off, and that guest did not. The first task was still running and held the lock, so the second one could not start. Homie had no hard stop and showed failures as plain text.

Fix

A stop action (hard power-off) that always needs the guest’s name typed, and the server refuses any action while the guest has an active task. Task results are green or red, and a failed shutdown suggests Stop.

The Public Demo

I wanted a link in the README that shows Homie on a real lab, without showing the real lab. RFC 0016 added roles:

• A role lists its pages (the demo gets Infrastructure, Workloads and GitOps), an explicit namespace allowlist (exact names, no globs, so a new namespace never shows up by accident), and Proxmox tags (only guests tagged demo; nodes show without addresses).
• A role never runs actions. Every route except reading queries, the live stream and logout answers 403, logged as msg="denied".
• Try the demo on the login page starts a session with the role, no password. Sessions last at most an hour, at most 200 at once, and get no live watches, so visitors never add load on kube-apiserver.

The filter in Homie is one boundary. Kubernetes is the second: the role reads the cluster through a ServiceAccount that only has view in the three allowed namespaces.

flowchart LR
  V[Visitor] --> CF["Cloudflare edge<br/>TLS, rate limit"] --> T[Tunnel] --> PG[public-gateway] --> D["homie-demo pod<br/>role demo"]
  D -- "impersonate" --> R["SA reader<br/>view in homie, databases, litellm"] --> K[kube-apiserver]
  D -- "PVEAuditor token" --> P[Proxmox]

I also ran it as a separate instance, not a role inside my own Homie. The demo pod holds only a read-only Proxmox token and a ServiceAccount that may impersonate exactly one other ServiceAccount, reader. No long-lived token exists for it. If the demo is ever compromised, that is all it reaches.

Issue 3: The Demo Showed Error Cards

Symptom

The first demo deploy showed an error instead of the Flux list, and an Argo CD error card on a cluster that has no Argo CD.

Cause

Both came from the role’s narrow RBAC. Homie listed GitRepositories for the Flux page, reader may not, and one failed list failed the whole page. For Argo CD, kube-apiserver checks authorization before it checks whether the resource exists, so reader got a 403 instead of the “not installed” answer Homie expected.

Fix

A role lists only the kinds it can be shown (Kustomizations and HelmReleases; sources were already hidden), and never queries Argo CD or Helm. That is also why reader needs no extra rules for them.

Issue 4: Flux Applied Only the Namespace

Symptom

The demo’s Flux Kustomization applied the namespace and then stopped.

Cause

My Kyverno policy restrict-externalsecret-paths (Part 11) lets a namespace read only OpenBao paths that start with its own name. The demo’s ExternalSecret read homie/demo and homie/app, under the main instance’s path.

Fix

Its own path, secret/homie-demo/app, with only the read-only Proxmox token and the password hash for the e2e login. The policy did its job: the demo should never have been able to read the main instance’s secrets.

Going Public

The tunnel already forwards *.franzramadhan.dev to public-gateway, and public-gateway only takes routes from namespaces labelled exposure=public. Publishing was four small changes:

• the namespace label exposure=public,
• an HTTPRoute for homie-demo.franzramadhan.dev on public-gateway with external-dns.kubernetes.io/cloudflare-proxied: "true", so external-dns creates the proxied CNAME to the tunnel,
• the Cilium network policy also allows the ingress entity, because traffic from the Gateway comes from Cilium’s Envoy, not from a pod,
• public_url set to the public host. Homie checks a login’s Origin header against public_url, so after this change the tailnet copy still serves the page but refuses logins with 403.

The demo lists only guests tagged demo, so the 8 k0s VMs got the tag. GitHub Actions was having trouble that evening, so I applied it on each host with qm and put the same change in Terraform, so the next plan shows no drift. qm set --tags replaces the whole list, so it gets the full list back plus the new tag:

t=$(qm config "$id" | sed -n 's/^tags: //p')
qm set "$id" --tags "$t;demo"

Issue 5: The Cloudflare Token Looked Invalid

Rate limits belong in front of Homie, not in it. The free plan includes one rate limiting rule, and I wanted it in place before the route went live:

(http.host eq "homie-demo.franzramadhan.dev"
 and (starts_with(http.request.uri.path, "/api/")
   or starts_with(http.request.uri.path, "/auth/")))
60 requests per 10 seconds per colo and IP, block for 10 seconds

Symptom

With a new API token, GET /user/tokens/verify answered Invalid API Token (code 1000), and GET /zones?name=franzramadhan.dev answered success: true with 0 results.

Cause

Two separate things. The token was account-owned, and those are verified at /accounts/<id>/tokens/verify, which said active. And the token did not have Zone Read on the zone yet, which the zones list reports as an empty result, not as an error.

Fix

Zone Read and Zone WAF Edit on the zone. The zone had no rate limit ruleset yet, so one PUT to /zones/<id>/rulesets/phases/http_ratelimit/entrypoint created it with the rule. A burst of 200 requests from my laptop, 25 at a time, got one 429 from Cloudflare. My burst only reached about 2.5 requests per second, so it rarely crossed the limit, but the rule is there.

Verifying It

From outside: / returns 200 and /api/meta returns 401 without a session. The demo part of the e2e script, pointed at the public URL:

PASS demo options ({"demo":true,"local":true,"oidc":false})
PASS demo login (204)
PASS demo namespaces (homie,litellm)
PASS demo refuses /api/q/alerts/lab (403)
PASS demo refuses /api/q/tailscale/_ (403)
PASS demo refuses /api/github/repos (403)
PASS demo refuses power (403)

The guest list shows exactly the 8 k0s VMs. databases shows no workloads, because the CloudNativePG pods have no Deployment or StatefulSet; that is a gap in Homie’s workload list, not in the role. The demo pod uses 2 MiB.

Outside 23:30 to 06:00 WIB the lab is off, so a visitor gets Cloudflare’s tunnel error page. A friendlier page would be a Cloudflare custom error page, not Homie’s job.

Lessons

1. Write the RFC even when you are the only reader. It took minutes, it decided what not to build, and the rejected ones stopped me from re-arguing them an hour later.
2. Test against the real thing early. The URL encoding bug and the stuck shutdown passed every unit test and failed on the first real VM.
3. Narrow RBAC changes what errors look like. A 403 can mean “you may not know whether this exists”, so a role must not ask questions it cannot be answered.
4. Two boundaries for anything public. The role filter in Homie and the ServiceAccount in Kubernetes each stop a bug in the other; the separate pod limits what a compromise reaches.
5. An empty answer is not always a no. Zero zones meant a missing permission, and an invalid token meant the wrong verify endpoint.

References

• GitHub: franzramadhan/homie
• GitHub: homie RFC 0001, foundation
• GitHub: homie RFC 0016, roles and a public demo
• GitHub: wheresryan22/anatomy
• Kubernetes: User impersonation
• Kubernetes: Structured authentication configuration
• Proxmox VE: User management and privileges
• Cloudflare: Rate limiting rules
• Cloudflare: Account-owned API tokens
• Cilium: Entities based policy