Back to Home
post-thumb

Homelab Rebuild Part 40: Taking Rate Limiting Back from Cloudflare

| October 28, 2026 | 18 min to read

Intro

Two apps in the lab are public: homie-demo and ipcalc. Their APIs sat behind one Cloudflare WAF rate limit rule, the only one the free plan gives me. Earlier today I wrote about dvarapala, the anti-DDoS gateway I built. The obvious next step was to put it in front of my own apps instead of the Cloudflare rule.

The rule is gone now. This post covers how dvarapala sits in the request path, what changed for someone who calls the API too fast, and how the two homie-demo replicas share their counts through Postgres.

TL;DR: each app has one dvarapala Deployment, and the HTTPRoute sends only /api/ (and /auth/ on homie-demo) through it. A token bucket of 60 with 6 back per second keeps the old steady rate of 60 per 10 s. But a visitor who comes in from idle now gets up to about 120 in the first 10 s. A throttled request waits 1 s instead of 10. homie-demo runs two replicas. They add their counts to a 10 s window in a dvarapala database on the CloudNativePG cluster, so a visitor cannot double their allowance by landing on both.

The Figure

Tokens left, the ledger count, and a 429 lampBarrier up: request let throughOne visitor's requests alternate between the two dvarapala replicas. Each booth's gauge is its own bucket for that visitor: 60 tokens, 6 back per second. Each second both booths add what they let through to the hopper, the count in Postgres for a 10 second window. Past 120 in a window (burst 60 plus 6 per second for 10 seconds) both booths turn the visitor away until the window ends, even if one still has tokens. Click, or press 1 to 5, to set the flow from 0 to 30 requests per second.

The figure is two tollbooths with a hopper between them. Each booth is a dvarapala replica. The gauge on its front is its own token bucket for one visitor: 12 bars of 5 tokens, refilled at 6 a second. The hopper is the shared count in Postgres. Every second (the lamp on top blinks), both booths add what they let through. The hopper gauge fills toward 120 for the current 10 second window, and the pips above it count the seconds. A barrier lifts for each request that goes through. When a booth’s bucket is empty, or the hopper passes 120, the barrier stays down and the lamp on the booth’s roof lights. That is a 429. Click the figure, or press 1 to 5, to change the flow from 0 to 30 requests per second. Built with the anatomy skill.

The Situation

  • Both apps are reached through a Cloudflare Tunnel, then the Cilium public-gateway in the k0s cluster (Part 9).
  • The old Cloudflare rule: 60 requests per 10 s per IP and Cloudflare data center, then a 10 s block, on homie-demo /api/ and /auth/ and ipcalc /api/. On the free plan, that is the whole feature: one rule, a 10 s period, a 10 s block. I had to add every public app to that one expression.
  • dvarapala 0.6.2, from ghcr.io/franzramadhan/dvarapala. Version 0.6.1 was the first with a container image, and 0.6.2 is the first image with the Postgres store compiled in (more on that below).
  • CloudNativePG cluster pg, two instances on NVMe data disks (Part 8), passwords from OpenBao through External Secrets (Part 5).

Setup

dvarapala does one upstream, the gateway does the routing

dvarapala proxies to exactly one upstream and has no host or path matching. It rate limits everything it receives. That decided the layout. Each app gets its own dvarapala Deployment and Service, and the HTTPRoute picks which paths go through it. Static pages and assets skip it.

flowchart LR
  V[Visitor] --> CF[Cloudflare edge] --> T[Tunnel] --> GW[Cilium public-gateway]
  GW -->|/api/, /auth/| D[dvarapala<br/>2 replicas]
  GW -->|everything else| H[homie-demo]
  D --> H
  D -. counts every 1 s .-> PG[(CNPG pg<br/>db dvarapala)]
rules:
  - matches:
      - path: {type: PathPrefix, value: /api/}
      - path: {type: PathPrefix, value: /auth/}
    backendRefs:
      - name: dvarapala
        port: 80
  - backendRefs:
      - name: homie-demo
        port: 80

The whole configuration is environment variables, so there is no ConfigMap. Lists are JSON:

env:
  - name: DVARAPALA__UPSTREAM
    value: http://homie-demo.homie-demo.svc.cluster.local
  - name: DVARAPALA__TRUSTED_PROXIES
    value: '["10.0.0.0/8", "192.168.101.0/24"]'
  - name: DVARAPALA__RATE_LIMIT__RATE
    value: "6"
  - name: DVARAPALA__RATE_LIMIT__BURST
    value: "60"
  # every peer is a gateway envoy: no per-peer connection cap
  - name: DVARAPALA__L4__EXEMPT
    value: '["10.0.0.0/8", "192.168.101.0/24"]'

The probes are tcpSocket, not HTTP. dvarapala proxies every HTTP request, so an HTTP probe tests the app behind it, not dvarapala. The pod runs as 10001 from a FROM scratch image with a read-only root filesystem. It requests 16 Mi and has 64 Mi as the limit. The CiliumNetworkPolicy lets it receive from the gateway. It can only talk to DNS, the app pods and, on homie-demo, Postgres on 5432.

Who gets counted

A rate limiter behind three proxies is only useful if it counts the visitor, not the last proxy. The request reaches dvarapala with an X-Forwarded-For header. Cloudflare started it, and every hop after it appended to it. dvarapala takes the rightmost address that is not in trusted_proxies. Pods are in 10.0.0.0/8 and the nodes in 192.168.101.0/24, so that address is the one Cloudflare saw. If a client writes an address into the header itself, it is further left, and dvarapala ignores it.

l4.exempt matters for a related reason. dvarapala also caps open connections per peer (max_per_client = 256 by default), and here every TCP peer is a gateway envoy. Without the exemption, the whole internet shares 256 connections.

I checked the keying through the real path instead of trusting the config. A pod inside the cluster sent 120 parallel requests to the public gateway with Host: ipcalc.franzramadhan.dev and a made-up X-Forwarded-For: 198.51.100.7:

# as 198.51.100.7
  63 x 404   (ipcalc's own answer, the test skipped Cloudflare's headers)
  57 x 429
# one request as 198.51.100.8 right after
   1 x 404

The Cilium envoy kept the header and appended its own trusted hop. So .7 had its own bucket, and .8 had a fresh one.

Shared counts in CloudNativePG

One replica is a single point of failure in front of the app. Two replicas that each keep their own memory give a visitor two buckets, and that doubles the limit. dvarapala solves that with a store (Redis or Postgres). Nothing on the request path waits for the store. Each replica keeps its local bucket. Every second, in the background, it adds the requests it allowed per client to a shared count. The count is for a 10 s window aligned to the clock. When a client’s count for the window passes burst + rate x 10 s (60 + 60 = 120), every replica throttles that client until the window ends.

I build the release binaries and the first image without the store features, because most people run one process. So the image now gets its own static musl build with --features redis,postgres (5.0 MB), released as 0.6.2. The tarballs and the Homebrew formula did not change.

On the database side, I use the same pattern as the other apps on pg. The Cluster spec manages the role, with its password from an ExternalSecret. A Database resource creates the database:

# Cluster pg, spec.managed.roles
- name: dvarapala
  ensure: present
  login: true
  passwordSecret:
    name: pg-dvarapala
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: dvarapala
spec:
  cluster:
    name: pg
  name: dvarapala
  owner: dvarapala

dvarapala creates its own tables at start, in a dvarapala schema, as UNLOGGED. PostgreSQL does not replicate them to the standby and empties them after a crash or failover. Counts that live for 10 s do not need to survive either event.

The password is in OpenBao twice: at databases/dvarapala for CNPG and at homie-demo/dvarapala for the app. The reason is the Kyverno policy. It only lets the ExternalSecrets of each namespace read their own path. The app side templates the whole connection string:

target:
  name: dvarapala-store
  template:
    data:
      url: "postgres://{{ .username }}:{{ .password }}@pg-rw.databases.svc:5432/dvarapala?sslmode=verify-full"

verify-full needs the CA. The image has no CA bundle at all (it is FROM scratch). So the lab root CA is a ConfigMap, mounted at /etc/lab-ca and set as DVARAPALA__STORE__CA_FILE. The Deployment has 2 replicas, a topology spread across nodes and a PodDisruptionBudget of minAvailable: 1.

The first rollout raced itself

Flux applied the role, the Database and the Deployment together. The Database reconciled before CNPG created the role:

role "dvarapala" does not exist (SQLSTATE 42704)

and the dvarapala pods started before the database existed:

msg="store unavailable" ... database "dvarapala" does not exist (3D000)

Neither needed a hand. CNPG retried the Database when the role was there. dvarapala kept serving on its local buckets (it fails open when the store is gone). Then it reconnected and logged store schema migrated version=1 and then store recovered. Both pods reported dvarapala_store_up 1 and dvarapala_store_replicas 2 within about 15 s.

ipcalc stays on one dvarapala replica with no store. It is a stateless calculator, and a single replica has exact counts. A store only adds a dependency there.

What Changed for Visitors

The steady rate is the same. The shape around it is not.

Cloudflare ruledvarapala
Counting60 per fixed 10 s period, per IP and Cloudflare data centertoken bucket per visitor IP: 60 to start, 6 back per second
Most requests from idle in the first 10 s60about 120 (60 burst plus 60 refilled)
Sustained flood60 per 10 s6 per second, the same 60 per 10 s
Over the limit429, retry-after: 10, blocked for 10 s429, retry-after: 1, the next token is 1/6 s away
Where the 429 comes fromCloudflare’s edgeinside the cluster, after the tunnel and the gateway
Two replicasnot applicableone shared allowance of 120 per 10 s window

A well-behaved client that retries after retry-after now waits 1 s instead of 10. A client that keeps sending at full speed still gets 6 requests a second and nothing more.

Here are the numbers from the real thing. While the Cloudflare rule still existed, I sent 200 parallel requests from my laptop to ipcalc’s /api/. They gave 93 x 200 and 104 x 429 from dvarapala (retry-after: 1). Only 3 got the Cloudflare retry-after: 10, so dvarapala already did nearly all the limiting. After I deleted the rule, I sent the same burst against homie-demo. It gave 54 x 429, all retry-after: 1. homie-demo itself answered the other 146 (a 404, because the test path does not exist). Not one came from Cloudflare.

The metrics showed the shared count at work:

# pod 1
dvarapala_requests_total{verdict="allow",protocol="http1"} 75
dvarapala_requests_total{verdict="throttle",protocol="http1"} 26
# pod 2
dvarapala_requests_total{verdict="allow",protocol="http1"} 72
dvarapala_requests_total{verdict="throttle",protocol="http1"} 18
dvarapala_requests_total{verdict="throttle",protocol="http1",scope="cluster"} 10

Pod 2 turned away those 10 requests with scope="cluster" while its own bucket still had tokens. The window count from both pods was over 120.

Sharing is approximate by design. A count reaches the other replica within two sync rounds. So the worst-case overshoot per window is replicas x rate x 2 x (sync interval + store timeout) = 2 x 6 x 2 x 1.5 s = 36 requests. For a demo app, that is a fair price. In return, the database is never on the request path.

What Cloudflare still does

Cloudflare still sits in front: TLS, the tunnel, its DDoS protection for floods at the network edge, and the offline page at night. Only the per-IP limit on the API paths moved into the cluster. There is a cost. An over-limit request now travels through the tunnel and the gateway before dvarapala refuses it. Before, the edge dropped it for free. At demo traffic, that cost is nothing. In a real attack it matters, and the edge protection is the layer for that.

Known Gap: No Throttle Logs, Metrics Nobody Reads

dvarapala logs nothing when it throttles a request. It only counts, and its metrics listen on 127.0.0.1:9090 inside the pod. Every number above came from a kubectl port-forward. Today nothing scrapes those metrics, and no alert fires on a burst of 429s. So I only notice an attack if I look. To get them into VictoriaMetrics with an alert on throttle spikes, I need a separate change.

I closed this gap two days later. See the update below.

Update 2026-10-30: the gap is closed

dvarapala now reports into the same VictoriaMetrics, VictoriaLogs and Grafana as everything else (Part 28), with its own dashboard and alerts. By now both apps run dvarapala 0.15.0 with a dvarapala.toml ConfigMap instead of the environment variables above. ipcalc runs one replica, and homie-demo runs two.

The logs were already there, just unreadable

The otel-agent DaemonSet collects the stdout of every pod. So the dvarapala lines were in VictoriaLogs from day one (about 57 an hour from homie-demo, 5 from ipcalc). Nobody was able to filter them. dvarapala writes coloured terminal output, and VictoriaLogs stored every line with its escape codes:

\u001b[2m2026-10-30T08:12:00.693808Z\u001b[0m \u001b[32m INFO\u001b[0m \u001b[2mdvarapala::admin::serve\u001b[0m ...

dvarapala logs through tracing_subscriber, which honours the NO_COLOR convention (dvarapala’s own tests set it). So the fix is one environment variable on the container, with no code change:

env:
  # plain key=value log lines in VictoriaLogs, no ANSI colour codes
  - name: NO_COLOR
    value: "1"
2026-10-30T08:28:40.388725Z  INFO dvarapala: proxy listening listen=0.0.0.0:8080 ...

Now a LogsQL word filter such as k8s.container.name:dvarapala ERROR matches what it says.

Metrics off the loopback

The metrics listener defaults to 127.0.0.1:9090. To move it to all interfaces, I added one TOML section and a container port named metrics:

# Prometheus endpoint, scraped by otel-cluster (job dvarapala); the
# network policy lets only that collector reach it
[metrics]
listen = "0.0.0.0:9090"

A listener on the pod IP is not public. The HTTPRoute only points at port 80 of the dvarapala Service. The CiliumNetworkPolicy gets one more ingress rule that lets only the cluster collector in:

- fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: monitoring
        app.kubernetes.io/instance: otel-cluster
  toPorts:
    - ports: [{port: "9090", protocol: TCP}]

The collector scrapes it like the other apps, by pod label and port name:

- job_name: dvarapala
  scrape_interval: 30s
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [homie-demo, ipcalc]
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name, __meta_kubernetes_pod_container_port_name]
      regex: dvarapala;metrics
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod

A minute after the pods rolled, VictoriaMetrics had up = 1 for all three pods and dvarapala_store_up = 1 on both homie-demo replicas.

A dashboard per question

Homelab / dvarapala in Grafana, a quiet afternoon on the demo

The dashboard has an instance switch (homie-demo, ipcalc or both), and each row answers one question:

RowQuestion
OverviewIs anyone being refused, and by what? Verdicts per second, refusals by scope (empty for the local bucket, cluster for the shared Postgres count, block for a manual block), per route, upstream errors, visitors in the limiter table, detection state
ChallengeDo visitors pass the proof of work on the homie-demo login, or fail it?
WAFWhat would the Coraza sidecar block in enforce? Verdicts, the top CRS rule IDs over an hour, inspect latency, agent errors
Shared storeIs the Postgres sync keeping up? Sync p95, replicas seen, errors
Console and adminWho logged in to the dvarapala console, and did a config reload fail?
ResourcesCPU and memory of dvarapala and the WAF sidecar against their limits
Logsdvarapala and coraza-spoa lines from VictoriaLogs, with a LogsQL filter box

I wanted the WAF row most. Both apps run the WAF in observe. The plan is to switch to enforce after some days without false positives. That decision now has a panel instead of a grep for waf match.

Seven alerts

They go to the same Telegram and email contact point as the rest (Part 32):

AlertFires when
upstream errorsmore than 0.1 failed upstream requests a second for 5 min: visitors get 502
store downdvarapala_store_up is 0 for 5 min: the homie-demo replicas fall back to their own buckets, so a visitor gets about twice the limit
WAF agent errorsany coraza-spoa failure in 10 min: requests pass with verdict error and nothing is inspected
attack detectedadaptive detection is in Attack for 2 min (it is in observe, so this tells me, it does not block)
config reload faileda reload was rejected: the old config keeps running, silently
console logins refused3 or more denied or failed console logins in 15 min
error logsmore than 5 ERROR lines in 10 min, from VictoriaLogs

There is no “dvarapala down” alert. The existing Metrics target down rule is max by (service_name, pod)(up) over every scrape job. So it already fires for a dvarapala pod that stops answering. A second rule only sends the same page twice.

The thresholds are a first guess at demo-sized traffic. I will check them against a few days of data.

Two Grafana surprises

The upstream errors summary used {{ humanize $values.A.Value }}. On the very first evaluation, before the series existed, Grafana logged:

error calling humanize: can't convert <nil> to float

humanize fails on a missing value. printf "%.2f" prints it without an error. The other trap: the threshold expression has gt, lt and ranges but no eq. So I wrote “detection state is 2 (Attack)” as gt 1.5 on max(dvarapala_detection_state).

Why XDP stays off

dvarapala can also run an XDP program that drops blocked sources and packet floods in the NIC driver (the dvarapala post has the details). I looked at turning it on here and left it off. The reason has nothing to do with dvarapala:

flowchart LR
  V[Visitor<br/>real IP] --> CF[Cloudflare edge]
  CF --> T[cloudflared<br/>outbound tunnel]
  T --> GW[gateway envoy<br/>10.0.0.0/8]
  GW -->|packet source = envoy| D[dvarapala pod]
  • No visitor packet ever reaches the lab. The tunnel is an outbound connection from cloudflared. So the packets that arrive at a dvarapala pod come from a gateway envoy, a pod or node address. The visitor’s IP only exists in X-Forwarded-For, which is L7. XDP keys its block list and packet rates on the packet’s source prefix. So it sees a handful of envoys and nobody else.
  • Enforce points the gun at the gateway. Those addresses are already in l4.exempt for the same reason. An XDP block or a packet rate on them cuts off every visitor at once.
  • It needs a different build and more privilege. The published image is built with --features redis,postgres, not xdp. The pod also needs CAP_BPF and CAP_NET_ADMIN, and the restricted pod security profile does not allow them.

XDP belongs on a host that receives the internet’s packets directly. Here that host belongs to Cloudflare, and its DDoS protection is the layer for packet floods.

Lessons

  1. Compare the whole curve, not the headline number. “60 per 10 s” fits both limiters, but it describes two different experiences. One is a fixed window with a 10 s block. The other is a bucket that allows a bigger first burst and then refuses only for a second at a time.
  2. Let the gateway route and the limiter limit. The single upstream in dvarapala looked like a limitation until the HTTPRoute did the path selection. Each piece stays simple.
  3. Test who the limiter thinks the client is, through the real path. If the trusted proxy list is one hop off, the limiter counts the gateway instead of the visitor. Then every visitor shares one bucket.
  4. Shared state can be approximate. Background sync, fail open and a known overshoot of 36 are better than a database call on every request.
  5. The Postgres you already run is enough for counters. It takes a managed role, a Database resource and unlogged tables. There is no Redis to add and operate.
  6. Check what the collector already has before adding a pipeline. The logs were in VictoriaLogs all along. The fix was NO_COLOR=1, not a new shipper.
  7. A packet-level defence needs the attacker’s packets. Behind a tunnel and a gateway, the source address on the wire is your own proxy. So XDP has nothing to tell apart. Put it where the internet’s packets land.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox
  24. Part 24: Pi-hole Without a Home Host
  25. Part 25: It Was the Router
  26. Part 26: TLS on the Front, Plaintext Behind It
  27. Part 27: One Bad Power Adapter, Two Different Deaths
  28. Part 28: Observability, Take Three: Making It Worth Looking At
  29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
  30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
  31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
  32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
  33. Part 33: The Firewall Held, the Tailnet Did Not
  34. Part 34: From Watching to Blocking, and a Record of Who Did What
  35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
  36. Part 36: The Fans Were Loud, the CPUs Were Idle
  37. Part 37: A Power Trip, a Search Domain and a Wildcard
  38. Part 38: An Autonomous AI Pentester in a Sealed Box
  39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
  40. Part 40: Taking Rate Limiting Back from Cloudflare (this post)

References