Back to Home
post-thumb

Homelab Rebuild Part 43: An AWS Emulator in k0s, and the Tailscale Rule That Broke One Pod in 256

Technology Kubernetes Homelab Cilium Networking
| November 13, 2026 | 11 min to read

Intro

I wanted tests to call AWS APIs without an AWS account. A job on the cluster’s GitHub runners can create a bucket, send a message to a queue, invoke a Lambda function, and throw it all away. floci is an AWS API emulator (a server that answers the AWS APIs like the real service) that runs as one container, so I put it in the k0s cluster.

S3 and SQS worked on the first try. Lambda did not. The function pod timed out while it downloaded its code from floci, but test pods with the same spec on the same node worked. The cause was a packet mark collision between two programs that both write to the same bits.

TL;DR: floci 2.4.0 runs in namespace floci, in memory, with Lambda as one pod per execution environment, and a LimitRange gives those pods the CPU request that Kyverno requires. Two things broke. A Flux GitRepository pinned to a commit made source-controller clone the whole floci history and run out of memory, so I vendored the chart. And host tailscaled masqueraded cross-node pod traffic for every Cilium identity whose low byte is 0x04, because Cilium writes the identity into the bits that Tailscale uses for its forward mark 0x40000/0xff0000. tailscale set --netfilter-mode=off on all eight VMs fixed it.

Pins that fall through a hole, the stamp, the leverKey plate: Tailscale's mark 0x04Each card is a packet leaving a pod for another node. Its holes are the pod's Cilium identity, which Cilium writes into bits 16 to 31 of the packet mark. The stencil hides the high byte, so the pins read bits 16 to 23, the bits under Tailscale's mask 0xff0000. When they read 0x04, the hammer stamps the card and ts-postrouting masquerades it to the node IP: identity 2052 (0x0804) was the Lambda function that could not reach floci. Click, or press space, to set netfilter mode off. Press n for the next card.

The figure is a card reader. Each card is a packet, and its punched holes are the packet mark. The stencil plate is Tailscale’s mask 0xff0000: it lets only bits 16 to 23 through. When those bits read 0x04, the hammer stamps the card SNAT and it lands in the wrong tray. Click the figure, or press space, to turn Tailscale’s netfilter rules off, and every card goes through untouched. Press n to feed the next card.

The Situation

  • Proxmox VE 9 on three hosts, eight Debian 13 VMs: three k0s controllers and five workers (k0s 1.36.5).
  • Cilium 1.20.4 with no kube-proxy, native routing, WireGuard pod encryption, legacy host routing and iptables masquerading.
  • Tailscale 1.104.5 on every VM itself, for SSH and cluster access. It ran with the default netfilter mode, so it managed its own iptables chains.
  • Flux 2.10.0 owns the cluster. Every app is a folder under kubernetes/apps/, and Kyverno 3.9.2 denies pods without CPU and memory requests.

Setup

floci has a Helm chart in its repo, charts/floci, but it is not published to a Helm repository. The app folder holds a namespace.yaml, a HelmRelease, a values.yaml that becomes a ConfigMap, and a LimitRange. These are the values that matter:

image:
  tag: "2.4.0"
floci:
  storage:
    mode: memory      # a restart deletes every bucket, table and queue
  lambda:
    executor: kubernetes
resources:
  requests: { cpu: 100m, memory: 256Mi }
  limits: { cpu: 500m, memory: 1Gi }

With the kubernetes executor, floci creates one pod per Lambda execution environment in its own namespace. A busybox init container, code-download, fetches the function’s code zip from the floci pod over HTTP. Then the AWS Lambda runtime image runs the handler.

The endpoint is in-cluster only, with no route from the LAN or the tailnet. floci accepts any credentials:

export AWS_ENDPOINT_URL=http://floci.floci.svc.cluster.local:4566
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
aws s3 mb s3://smoke && aws s3 ls

Services that floci runs as Docker containers, such as RDS and ElastiCache, do not work here. The workers run containerd and have no Docker socket. A test that needs Postgres uses the cluster’s CloudNativePG instance instead.

A LimitRange for the Lambda pods

The chart’s pod template showed that floci sets only memory on the Lambda pods: the request and the limit both equal the function’s MemorySize. The cluster’s Kyverno policy audit-workload-hygiene runs in Deny mode and requires a CPU request on every container, so it can reject every Lambda pod. A LimitRange in namespace floci fills in what the pods leave out. The API server applies it during admission, before Kyverno validates the pod:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-resources
  namespace: floci
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 50m, memory: 64Mi }
      default: { cpu: 500m, memory: 256Mi }

The memory defaults are for the busybox init container, which floci leaves without resources.

Issue 1: Flux Lost Every Source

Symptom

A few minutes after the first merge, every Flux GitRepository and HelmRepository in the cluster stopped updating. source-controller restarted 16 times in about 15 minutes, and each time it reached its 256Mi limit and failed its liveness probe.

Cause

The chart landed on floci main on 2026-11-09, after the 2.4.0 tag. So I pinned the GitRepository to a commit:

spec:
  url: https://github.com/floci-io/floci
  ref:
    commit: a65def00a4c9a94c8b759d5147b1d566ddd5136c

For a tag or a branch, source-controller does a shallow clone (one commit, no history). For a commit ref, it cloned the whole floci history, 113 MB, and ran out of memory. One controller serves all sources, so every app in the cluster lost its updates with it.

Fix

First I stopped the loop. Suspending only the floci objects was enough:

flux suspend kustomization floci -n flux-system
flux suspend source git floci -n floci
flux suspend helmrelease floci -n floci
# source-controller back to 98Mi, no more restarts

Then I copied charts/floci at that commit into kubernetes/apps/floci/chart/, with floci’s MIT LICENSE and without its ci/ and tests/ folders. The HelmRelease reads the chart from the cluster’s own repo:

chart:
  spec:
    chart: ./kubernetes/apps/floci/chart
    reconcileStrategy: Revision
    sourceRef:
      kind: GitRepository
      name: flux-system
      namespace: flux-system

When a floci release tag ships the chart, a GitRepository on that tag with sparseCheckout replaces the copy. After the fix, source-controller sits at about 147Mi.

Issue 2: The Lambda Pods Could Not Reach floci

Symptom

I created a function called smoke and invoked it. floci created a Lambda pod on k0s-pve-01-worker-2, and its init container code-download timed out on its wget to the floci pod IP. floci deleted the pod, created a new one about a minute later, and failed again:

WARN  [LambdaExecutorService] Failed to acquire container for function smoke:
  Pod floci-lambda-smoke-45e0368a for function 'smoke' failed to start:
  code-download: exited with code 1

A shell in a pod with the same labels on the same worker showed the same thing:

Connecting to 10.0.3.101:4566 (10.0.3.101:4566)
wget: download timed out
nc: 10.0.3.101 (10.0.3.101:4566): Connection timed out

Dead ends

Each of these looked like a cause and was not:

  1. A network policy. The namespace has none.
  2. The node. Test pods on three workers, including k0s-pve-01-worker-2, reached floci.
  3. The init container. A test pod with the same busybox:1.36 wget in an init container and in the main container passed both.
  4. Timing at pod start. Twelve wget tries in a row from a new pod all passed.
  5. Cilium’s ipcache (its table of which IP belongs to which identity). Both nodes had the right /32 entries for the two pods.
  6. The pod spec. A copy of the failing pod’s spec worked.

All the test pods had my own labels, and that was the clue I missed at first. Cilium gives each set of pod labels a security identity (a number that its policies use instead of IP addresses). My test pods had different identities from the Lambda pod, so they did not test the same thing.

Finding it

Hubble (Cilium’s flow log) on the floci node showed the SYN and its retries, but not from the Lambda pod:

192.168.101.41:57644 (remote-node) -> floci/floci-594b45fc68-t4f8m:4566 (ID:20596) to-endpoint FORWARDED (TCP Flags: SYN)
192.168.101.41:57644 (remote-node) -> floci/floci-594b45fc68-t4f8m:4566 (ID:20596) to-endpoint FORWARDED (TCP Flags: SYN)

The source was the node IP of the worker that ran the Lambda pod, labeled remote-node, and no reply ever reached the pod. Something on that worker rewrote the source address. Cilium masquerades only traffic to destinations outside 10.0.0.0/8, so the rewrite was not Cilium’s. On that worker, the Tailscale NAT chain had counted it:

iptables -t nat -L ts-postrouting -v -n
# Chain ts-postrouting (1 references)
#  pkts bytes target     prot opt in  out  source     destination
#     7   420 MASQUERADE  all  --  *   *    0.0.0.0/0  0.0.0.0/0    mark match 0x40000/0xff0000

Seven packets matched the Tailscale masquerade rule. The chain that sets that mark for real Tailscale traffic, ts-forward, had counted 0.

Cause

Two programs use the same bits of skb->mark, the packet mark that the kernel carries with each packet:

  • Cilium writes the source pod’s security identity into the upper 16 bits when it hands a packet to the host network stack. With legacy host routing, cross-node pod traffic goes through the host stack and its iptables chains.
  • Tailscale, in its default netfilter mode, marks packets that it forwards from tailscale0 with 0x40000 under the mask 0xff0000. In POSTROUTING it masquerades every packet with that mark to the node IP.

The function smoke had identity 2052, which is 0x0804. Shifted into the upper 16 bits, that is mark 0x08040000. Under Tailscale’s mask, the mark reads 0x040000, which is the Tailscale forward mark:

flowchart LR
  A["Lambda pod<br/>identity 2052 = 0x0804"] --> B["Cilium writes mark<br/>0x0804 in bits 16 to 31"]
  B --> C{"mark AND 0xff0000<br/>is 0x040000?"}
  C -- "yes" --> D["ts-postrouting<br/>MASQUERADE to node IP"]
  C -- "no" --> E["pod IP kept"]
  D --> F["floci sees remote-node<br/>no reply reaches the pod"]
  E --> G["code download works"]

So every identity whose low byte is 0x04 (260, 516, 772, … 2052, …) loses its cross-node pod traffic. That is one identity in 256. Traffic to a pod on the same node does not go through the host’s iptables, so it works. Traffic between other identities works. Pods with the same labels always get the same identity, so the failure followed one function and looked random from the outside.

I already met this overlap once. In Part 3, the Cilium Gateway broke on these same VMs, and Tailscale’s 0xff0000 mark rules looked like a perfect collision with the ingress identity. There it was a dead end, and the real cause was src_valid_mark. Here the collision is real.

To prove it, I gave the same code a function named hello. It got identity 8297, 0x2069, whose low byte is 0x69. The cold invoke took 18.9 s (pod create, image pull, code download) and a warm invoke took 4.2 s, and both returned the payload.

Fix

The VMs do not route anything for the tailnet. They advertise no subnets and are not exit nodes, so tailscaled does not need its iptables rules at all. I turned them off, one VM at a time:

tailscale set --netfilter-mode=off
# Warning: netfilter=off; configure iptables yourself.
iptables -t nat -S | grep -c ts-
# 0
iptables -S | grep -c ts-
# 0
tailscale status --self --peers=false | head -1
# 100.x.y.z  k0s-pve-01-worker-1  ...  linux  -

I started with the worker that ran the failing pod, then the other four workers, and the three controllers last. Each VM stayed on the tailnet. After the last one, all five workers were Ready, Cilium reported 5/5 nodes reachable, and every Flux Kustomization and HelmRelease was Ready. The VM setup docs now use tailscale up --ssh --netfilter-mode=off, so a rebuilt VM gets the same setting.

The setting has a cost. With netfilter off, Tailscale also removes its rule that drops packets from 100.64.0.0/10 that do not arrive on tailscale0, which stops other interfaces from spoofing tailnet addresses. Nothing on the LAN uses that range, so I accepted it.

Lessons

  1. Pin Flux sources to tags, not commits. A commit ref makes source-controller clone the full history, and one large repo can take down every source in the cluster.
  2. Read what an operator puts in its pods. floci creates pods I cannot edit. Its template showed no CPU request, and a LimitRange filled it in before the policy saw a single pod.
  3. A copy of a failing pod is not the same pod. New labels give a new Cilium identity, and the identity is part of the packet.
  4. Read the masquerade counters. When Hubble shows remote-node as the source of pod traffic, iptables -t nat -L -v -n shows which rule rewrote it.
  5. Two programs, one mark. Cilium and Tailscale both use bits 16 to 23 of skb->mark. On a Cilium node that runs tailscaled only for access, set --netfilter-mode=off.

References