Intro
After Part 2 I could reach the cluster only through an SSH bastion. My Mac
could not talk to the LAN from tools like kubectl at all. I wanted the opposite:
every admin path over Tailscale, nothing exposed on the internet, and no
long-lived cluster credentials on my laptop.
TL;DR: I use Tailscale SSH for nodes and Tailscale Ingresses for admin UIs. For
kubectl, I use the API server proxy of the Tailscale Kubernetes operator in auth mode (the kubeconfig hastoken: unused). One gotcha: the Tailscale proxy only speaks TLS to backends on port 443.
The Situation
- 8 Debian 13 VMs with Tailscale installed but not logged in (Part 2)
- Tailscale operator chart 1.102.4
- A tailnet with MagicDNS and HTTPS certificates enabled
- The devices of the old cluster still registered in the tailnet
Setup
Tailscale SSH on every node
I joined each VM with SSH enabled and without accepting subnet routes. That way the VMs do not reroute LAN traffic that k0s and Cilium depend on:
tailscale up --ssh --hostname=$(hostname)
The default ssh rule of the tailnet policy (autogroup:member to
autogroup:self, check mode) already covers devices I own. So this works
with no key on the client:
ssh root@k0s-controller-01
# # Authentication checked with Tailscale SSH.
The kubeconfig with no secret
The Tailscale operator can run an API server proxy. In auth mode
(apiServerProxyConfig.mode: "true"), the operator impersonates the tailnet
identity of the caller. So the kubeconfig carries no token or certificate.
Permissions come from a grant in the tailnet policy:
{
"src": ["group:admin"],
"dst": ["tag:k8s-operator"],
"app": {
"tailscale.com/cap/kubernetes": [
{"impersonate": {"groups": ["system:masters"]}}
]
}
}
The OAuth client of the operator needs the devices:core, auth_keys and
services scopes and the tag:k8s-operator tag. Its credentials go into a
Kubernetes Secret that I create by hand, never into git. Then:
tailscale configure kubeconfig tailscale-operator
kubectl auth whoami
# # Username me@example.com
# # Groups [system:masters system:authenticated]
This also solved the Mac problem from Part 2: kubectl talks to a 100.x
address, which works for every binary.
Admin UIs as Tailscale Ingresses
OpenBao (Part 5) and Headlamp get ingressClassName: tailscale. This gives
each one its own https://<name>.<tailnet>.ts.net with a certificate from
Tailscale. Headlamp logins use short-lived tokens:
kubectl -n headlamp create token headlamp-admin --duration=12h | pbcopy
Issue 1: Every Device Name Ended in -1
Symptom
The new VMs showed up as k0s-controller-01-1, k0s-pve-01-worker-1-1, and so
on.
Cause
The devices of the old cluster were still registered and offline, and they held the names.
Fix
Delete the stale devices (I used a one-day API access token, then revoked it)
and rename the new ones. The same thing happened later with the operator itself
(tailscale-operator-1).
Issue 2: The Ingress Returned 400
Symptom
After OpenBao switched to TLS-only (Part 6), its Tailscale Ingress returned
400 on every request.
Cause
The Tailscale proxy sent plain HTTP to the HTTPS listener of OpenBao on port 8200.
It only uses TLS toward a backend when the Service port is 443. A port named
https is not enough.
Fix
Add a small Service that maps 443 to 8200 and selects the active pod. Only
the Ingress uses it:
apiVersion: v1
kind: Service
metadata:
name: openbao-tailscale
spec:
selector:
app.kubernetes.io/name: openbao
component: server
openbao-active: "true"
ports:
- name: https
port: 443
targetPort: 8200
Issue 3: The Chart Gave Headlamp Cluster-Admin
Symptom
When I applied my own ClusterRoleBinding named headlamp-admin, it printed a
warning that the object already existed.
Cause
By default, the Headlamp chart creates a ClusterRoleBinding called
headlamp-admin. It binds the pod of Headlamp itself to cluster-admin. Headlamp
does not need that when users bring their own token.
Fix
Set clusterRoleBinding.create: false in the chart values. Add a separately named
binding (headlamp-user-admin) for the login identity.
Update 2026-10-19: The allow-all grant
Next to the Kubernetes grant above, the policy kept the default Tailscale
allow-all grant, {"src": ["*"], "dst": ["*"], "ip": ["*"]}. That was fine
while every device on the tailnet was mine. It stopped being fine for one VM
I host for a partner. Over the tailnet it could reach the cluster API, the
nodes and my laptop, even though the Proxmox firewall kept it off the LAN.
The src now lists autogroup:member and the two Kubernetes tags. The VM is
tag:sandbox, and policy tests deny the sandbox any path in. The full story
is in Part 33.
Lessons
- Prefer identity over secrets. The tailnet already knows who I am, so the kubeconfig does not need to.
- Clean up old devices before reusing names. Tailscale does not reuse a name that an offline device still holds.
- Read what a chart creates by default. “Admin UI” charts tend to grant themselves admin.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet (this post)
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed