Back to Home

Homelab Rebuild Part 4: Living on the Tailnet

| October 07, 2026 | 6 min to read

Intro

After Part 2 I could reach the cluster only through an SSH bastion. My Mac could not talk to the LAN from tools like kubectl at all. I wanted the opposite: every admin path over Tailscale, nothing exposed on the internet, and no long-lived cluster credentials on my laptop.

TL;DR: I use Tailscale SSH for nodes and Tailscale Ingresses for admin UIs. For kubectl, I use the API server proxy of the Tailscale Kubernetes operator in auth mode (the kubeconfig has token: unused). One gotcha: the Tailscale proxy only speaks TLS to backends on port 443.

The Situation

  • 8 Debian 13 VMs with Tailscale installed but not logged in (Part 2)
  • Tailscale operator chart 1.102.4
  • A tailnet with MagicDNS and HTTPS certificates enabled
  • The devices of the old cluster still registered in the tailnet

Setup

Tailscale SSH on every node

I joined each VM with SSH enabled and without accepting subnet routes. That way the VMs do not reroute LAN traffic that k0s and Cilium depend on:

tailscale up --ssh --hostname=$(hostname)

The default ssh rule of the tailnet policy (autogroup:member to autogroup:self, check mode) already covers devices I own. So this works with no key on the client:

ssh root@k0s-controller-01
# # Authentication checked with Tailscale SSH.

The kubeconfig with no secret

The Tailscale operator can run an API server proxy. In auth mode (apiServerProxyConfig.mode: "true"), the operator impersonates the tailnet identity of the caller. So the kubeconfig carries no token or certificate. Permissions come from a grant in the tailnet policy:

{
  "src": ["group:admin"],
  "dst": ["tag:k8s-operator"],
  "app": {
    "tailscale.com/cap/kubernetes": [
      {"impersonate": {"groups": ["system:masters"]}}
    ]
  }
}

The OAuth client of the operator needs the devices:core, auth_keys and services scopes and the tag:k8s-operator tag. Its credentials go into a Kubernetes Secret that I create by hand, never into git. Then:

tailscale configure kubeconfig tailscale-operator
kubectl auth whoami
# # Username    me@example.com
# # Groups      [system:masters system:authenticated]

This also solved the Mac problem from Part 2: kubectl talks to a 100.x address, which works for every binary.

Admin UIs as Tailscale Ingresses

OpenBao (Part 5) and Headlamp get ingressClassName: tailscale. This gives each one its own https://<name>.<tailnet>.ts.net with a certificate from Tailscale. Headlamp logins use short-lived tokens:

kubectl -n headlamp create token headlamp-admin --duration=12h | pbcopy

Issue 1: Every Device Name Ended in -1

Symptom

The new VMs showed up as k0s-controller-01-1, k0s-pve-01-worker-1-1, and so on.

Cause

The devices of the old cluster were still registered and offline, and they held the names.

Fix

Delete the stale devices (I used a one-day API access token, then revoked it) and rename the new ones. The same thing happened later with the operator itself (tailscale-operator-1).

Issue 2: The Ingress Returned 400

Symptom

After OpenBao switched to TLS-only (Part 6), its Tailscale Ingress returned 400 on every request.

Cause

The Tailscale proxy sent plain HTTP to the HTTPS listener of OpenBao on port 8200. It only uses TLS toward a backend when the Service port is 443. A port named https is not enough.

Fix

Add a small Service that maps 443 to 8200 and selects the active pod. Only the Ingress uses it:

apiVersion: v1
kind: Service
metadata:
  name: openbao-tailscale
spec:
  selector:
    app.kubernetes.io/name: openbao
    component: server
    openbao-active: "true"
  ports:
    - name: https
      port: 443
      targetPort: 8200

Issue 3: The Chart Gave Headlamp Cluster-Admin

Symptom

When I applied my own ClusterRoleBinding named headlamp-admin, it printed a warning that the object already existed.

Cause

By default, the Headlamp chart creates a ClusterRoleBinding called headlamp-admin. It binds the pod of Headlamp itself to cluster-admin. Headlamp does not need that when users bring their own token.

Fix

Set clusterRoleBinding.create: false in the chart values. Add a separately named binding (headlamp-user-admin) for the login identity.

Update 2026-10-19: The allow-all grant

Next to the Kubernetes grant above, the policy kept the default Tailscale allow-all grant, {"src": ["*"], "dst": ["*"], "ip": ["*"]}. That was fine while every device on the tailnet was mine. It stopped being fine for one VM I host for a partner. Over the tailnet it could reach the cluster API, the nodes and my laptop, even though the Proxmox firewall kept it off the LAN. The src now lists autogroup:member and the two Kubernetes tags. The VM is tag:sandbox, and policy tests deny the sandbox any path in. The full story is in Part 33.

Lessons

  1. Prefer identity over secrets. The tailnet already knows who I am, so the kubeconfig does not need to.
  2. Clean up old devices before reusing names. Tailscale does not reuse a name that an offline device still holds.
  3. Read what a chart creates by default. “Admin UI” charts tend to grant themselves admin.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet (this post)
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths
  28. Part 28: Observability, Take Three: Making It Worth Looking At
  29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
  30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
  31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
  32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
  33. Part 33: The Firewall Held, the Tailnet Did Not
  34. Part 34: From Watching to Blocking, and a Record of Who Did What
  35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
  36. Part 36: The Fans Were Loud, the CPUs Were Idle
  37. Part 37: A Power Trip, a Search Domain and a Wildcard
  38. Part 38: An Autonomous AI Pentester in a Sealed Box
  39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References