Loading

Back to Home

Homelab Rebuild Part 4: Living on the Tailnet

Technology Kubernetes Homelab Tailscale
| October 07, 2026 | 4 min to read

Intro

After Part 2 I could reach the cluster only through an SSH bastion, and my Mac couldn’t talk to the LAN from tools like kubectl at all. I wanted the opposite: every admin path over Tailscale, nothing exposed on the internet, and no long-lived cluster credentials on my laptop.

TL;DR: Tailscale SSH for nodes, the Tailscale Kubernetes operator’s API server proxy in auth mode for kubectl (the kubeconfig has token: unused), and Tailscale Ingresses for admin UIs. One gotcha: the Tailscale proxy only speaks TLS to backends on port 443.

The Situation

  • 8 Debian 13 VMs with Tailscale installed but not logged in (Part 2)
  • Tailscale operator chart 1.102.4
  • A tailnet with MagicDNS and HTTPS certificates enabled
  • The old cluster’s devices still registered in the tailnet

Setup

Tailscale SSH on every node

I joined each VM with SSH enabled and without accepting subnet routes, so the VMs don’t reroute LAN traffic that k0s and Cilium depend on:

tailscale up --ssh --hostname=$(hostname)

The tailnet policy’s default ssh rule (autogroup:member to autogroup:self, check mode) already covers devices I own, so this works with no key on the client:

ssh root@k0s-controller-01
# # Authentication checked with Tailscale SSH.

The kubeconfig with no secret

The Tailscale operator can run an API server proxy. In auth mode (apiServerProxyConfig.mode: "true"), the operator impersonates the caller’s tailnet identity, so the kubeconfig carries no token or certificate. Permissions come from a grant in the tailnet policy:

{
  "src": ["group:admin"],
  "dst": ["tag:k8s-operator"],
  "app": {
    "tailscale.com/cap/kubernetes": [
      {"impersonate": {"groups": ["system:masters"]}}
    ]
  }
}

The operator’s OAuth client needs the devices:core, auth_keys and services scopes and the tag:k8s-operator tag. Its credentials go into a Kubernetes Secret created by hand, never into git. Then:

tailscale configure kubeconfig tailscale-operator
kubectl auth whoami
# # Username    [email protected]
# # Groups      [system:masters system:authenticated]

This also solved the Mac problem from Part 2: kubectl talks to a 100.x address, which works for every binary.

Admin UIs as Tailscale Ingresses

OpenBao (Part 5) and Headlamp get ingressClassName: tailscale, which gives each its own https://<name>.<tailnet>.ts.net with a certificate from Tailscale. Headlamp logins use short-lived tokens:

kubectl -n headlamp create token headlamp-admin --duration=12h | pbcopy

Issue 1: Every Device Name Ended in -1

Symptom

The new VMs showed up as k0s-controller-01-1, k0s-pve-01-worker-1-1, and so on.

Cause

The old cluster’s devices were still registered and offline, holding the names.

Fix

Delete the stale devices (I used a one-day API access token, then revoked it) and rename the new ones. Same thing happened later with the operator itself (tailscale-operator-1).

Issue 2: The Ingress Returned 400

Symptom

After OpenBao switched to TLS-only (Part 6), its Tailscale Ingress returned 400 on every request.

Cause

The Tailscale proxy sent plain HTTP to OpenBao’s HTTPS listener on port 8200. It only uses TLS toward a backend when the Service port is 443; naming the port https is not enough.

Fix

A small Service that maps 443 to 8200 and selects the active pod, used only by the Ingress:

apiVersion: v1
kind: Service
metadata:
  name: openbao-tailscale
spec:
  selector:
    app.kubernetes.io/name: openbao
    component: server
    openbao-active: "true"
  ports:
    - name: https
      port: 443
      targetPort: 8200

Issue 3: The Chart Gave Headlamp Cluster-Admin

Symptom

Applying my own ClusterRoleBinding named headlamp-admin printed a warning that the object already existed.

Cause

The Headlamp chart creates a ClusterRoleBinding called headlamp-admin by default, binding Headlamp’s own pod to cluster-admin. Headlamp doesn’t need that when users bring their own token.

Fix

clusterRoleBinding.create: false in the chart values, and a separately named binding (headlamp-user-admin) for the login identity.

Lessons

  1. Prefer identity over secrets. The tailnet already knows who I am; the kubeconfig doesn’t need to.
  2. Clean up old devices before reusing names. Tailscale won’t reuse a name that an offline device still holds.
  3. Read what a chart creates by default. “Admin UI” charts tend to grant themselves admin.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet (this post)
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers

References