Homelab Rebuild Part 33: The Firewall Held, the Tailnet Did Not

Intro

One VM on pve-01 is not mine. I host it for a partner. It lives outside the cluster, it is off overnight like the rest of the lab, and the partner reaches it over Tailscale. When I set it up, I put a Proxmox firewall group on its NIC. The goal was to stop it from talking to anything else in my home network.

This morning I wanted to make sure that it really was not able to. I expected a short “yes, it is isolated” and got half of one.

TL;DR: The Proxmox firewall held. From inside the VM, every LAN host was closed. But the VM was also on my tailnet, logged in as my own user. My policy still had the default allow-all grant, "src": ["*"]. Over Tailscale, the VM was able to open the cluster API, SSH on the nodes, Grafana, LiteLLM and my laptop. Tagging the VM did not help on its own, because "*" matches tagged devices too. The fix was a tag:sandbox tag together with an allow-all grant whose src lists who can connect. Policy tests now fail the save if the sandbox ever gets a path back. The share to the partner kept working.

The Figure

The figure is my lab drawn as a house with a guest annex. The main house is the homelab: the Proxmox hosts at the back, the k0s controllers and workers on the bench, Grafana on the wall, LiteLLM, and my laptop on the desk. The annex is the partner’s guest VM, and its street door (the internet) is open both ways. The party wall between them is the LAN. The Proxmox firewall group block-local bricks up its old doorway, with a small slot left for DNS to the router. That wall held. The corridor is the tailnet, and that is where it leaked. Press B, T or A (or the buttons under the card) to switch between before, tag only and after. Click a wall, a door, the sign or the visitor pass to list which probes get through. A tag:sandbox sign on the annex changes nothing, because "*" in a grant also matches tagged devices. After the fix, the corridor door only opens from the house side, and the partner reaches the annex with a visitor pass (a node share) from their own tailnet. I built it with the anatomy skill (see References).

The Situation

• Proxmox VE 9.2 on three hosts, the guest VM on pve-01, bridged on vmbr0 with the Proxmox firewall on (firewall=1).
• Tailscale 1.102.2 in the VM. The partner reaches it through a node share to their own account.
• The tailnet policy from Part 4: one allow-all IP grant and the Kubernetes operator grant. It is edited in the admin console, not in git.
• The k0s nodes, Grafana, LiteLLM and Headlamp are all on the same tailnet (Parts 4, 28 and 29).

I wanted to test from the point of view of the VM without logging in to it. So I used the QEMU guest agent from the host and bash’s /dev/tcp:

qm guest exec <vmid> --timeout 8 -- \
  timeout 4 bash -c 'echo > /dev/tcp/<host>/<port> && echo OPEN || echo CLOSED'

Issue 1: Is the LAN Really Walled Off?

Symptom

None, and that is the point of checking. The firewall file of the VM only pulls in a cluster-wide group:

# /etc/pve/firewall/<vmid>.fw
[OPTIONS]
enable: 1
[RULES]
GROUP block-local

Cause

The group in cluster.fw allows DNS to the router and drops the home ranges (two more ranges like the last line are left out here):

[group block-local]
IN ACCEPT -log nolog
OUT DROP -dest 192.168.101.200 -p tcp -dport 80 -log nolog
OUT DROP -dest 192.168.101.200 -p tcp -dport 443 -log nolog
OUT ACCEPT -dest 192.168.101.200/32 -log nolog
OUT DROP -dest 192.168.101.0/24 -log nolog

Proxmox checks rules from top to bottom. So the router answers DNS but not its admin pages, and the firewall drops the rest of the subnet.

Fix

Nothing to fix. From inside the VM:

Dead end: IPv6

Every rule above is IPv4. If the VM had a path over IPv6, the group does not see it. It has no global IPv6 address on its LAN interface. I sent a ping to all nodes on the link (ping -6 ff02::1%ens18), and only the VM itself answered. No IPv6 path today. I will repeat this test if the router ever starts handing out IPv6.

Issue 2: The Tailnet Was a Second Door

Symptom

tailscale status in the VM listed every machine I own: the Proxmox hosts (each one offers an exit node), all eight k0s nodes, the Tailscale operator’s devices and my laptop. The owner column showed my user, not a tag. So I tried the same probes over tailnet addresses:

The firewall group did its job on this traffic too, and it did not matter. Tailscale first tries a direct WireGuard path to the peer’s LAN address, and block-local drops it. It then falls back to a DERP relay on the internet (tailscale status showed relay "sin" for the worker). UDP to the internet is allowed. The tunnel came up anyway.

Cause

Two things together:

1. The VM was logged in to Tailscale as my user. In the policy, it was one of my devices, as trusted as my laptop.

2. The policy still had the default grant:

   {"src": ["*"], "dst": ["*"], "ip": ["*"]}
   

Every service on that list still had its own login (Tailscale SSH check mode, the API server proxy’s identity, SSO in front of Grafana and LiteLLM). But the network layer let anyone with a shell on that VM knock on all of them. An exit node on a Proxmox host can also forward such traffic onto the LAN. I did not test that one.

Dead end: Just tag it

The obvious move is a tag. The tag owns a tagged device, not me, so it stops being “one of my devices”. But in a grant, "*" means every device in the tailnet, tagged or not. With the allow-all grant unchanged, tag:sandbox gets exactly the same access as before.

Fix

In the policy, add a tag owner for the sandbox. Replace "*" with a src that lists who can open connections:

"tagOwners": {
    "tag:k8s-operator": [],
    "tag:k8s":          ["tag:k8s-operator"],
    "tag:sandbox":      ["autogroup:admin"],
},

"grants": [
    // no "*" src: it would include tag:sandbox
    {"src": ["autogroup:member", "tag:k8s", "tag:k8s-operator"], "dst": ["*"], "ip": ["*"]},
    // ... the Kubernetes operator grant from Part 4, unchanged
],

autogroup:member is every user-owned device, so my laptop and phone keep everything. "*" already covered the two Kubernetes tags, so listing them keeps their access the same. tag:sandbox is in no src. Other devices can reach the VM, but it cannot start a connection to anything on the tailnet, including exit nodes.

Then tests. The admin console runs them on every save:

"tests": [
    {
        "src":    "<me>",
        "accept": ["<laptop>:443", "<guest vm>:22"],
    },
    {
        "src":  "tag:sandbox",
        "deny": ["<controller>:6443", "<controller>:22", "<laptop>:22", "<grafana>:443"],
    },
],

If a later edit puts "*" back, the save fails. So the sandbox cannot get its access back without me noticing.

The policy went first, then the tag (Machines, the VM, Edit ACL tags, tag:sandbox). In that order there is no moment where the VM is tagged while "*" still lets it in, and the tests already pass before the tag exists.

After

The same probes from inside the VM:

The share survived the tag. The device still shows “Shared out”, and the partner connects from their own tailnet. Shared access comes from the share, not from my grants. My only loss: Tailscale SSH to the VM no longer matches my ssh rule (dst: autogroup:self), because the VM is no longer mine. Plain ssh over its tailnet address still works.

flowchart LR
  P[Partner's tailnet] -- share --> G[Guest VM<br/>tag:sandbox]
  M[My devices<br/>autogroup:member] --> G
  G -- internet only --> I[Internet]
  G -. blocked .-> L[LAN<br/>block-local]
  G -. no grant .-> T[Cluster, Grafana,<br/>LiteLLM, laptop]

Lessons

1. A host firewall does not see the overlay. WireGuard to a peer looks like internet traffic to the bridge. Any VM that can run Tailscale needs its isolation in the tailnet policy too.
2. Test from inside the thing you are isolating. qm guest exec plus /dev/tcp gave a real answer in a minute. Reading the rules told me only half of it.
3. "*" in a grant includes tags. A tag changes who owns a device, not what the allow-all grant lets it do. List the sources instead.
4. Never log a guest device in as yourself. It becomes one of your devices. Tag it at login or right after.
5. Put the boundary in policy tests. A deny test turns “I hope nobody adds * back” into a save that fails.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not (this post)
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• GitHub: wheresryan22/anatomy
• Tailscale: Grants
• Tailscale: Tags
• Tailscale: Autogroups
• Tailscale: Sharing your nodes with other users
• Tailscale: Tests in the tailnet policy file
• Proxmox VE: Firewall