Intro
Two apps in the lab are public: homie-demo and ipcalc. Their APIs sat behind one Cloudflare WAF rate limit rule, the only one the free plan gives me. Earlier today I wrote about dvarapala, the anti-DDoS gateway I built. The obvious next step was to put it in front of my own apps instead of the Cloudflare rule.
The rule is gone now. This post covers how dvarapala sits in the request path, what changed for someone who calls the API too fast, and how the two homie-demo replicas share their counts through Postgres.
TL;DR: each app has one dvarapala Deployment, and the HTTPRoute sends only
/api/(and/auth/on homie-demo) through it. A token bucket of 60 with 6 back per second keeps the old steady rate of 60 per 10 s. But a visitor who comes in from idle now gets up to about 120 in the first 10 s. A throttled request waits 1 s instead of 10. homie-demo runs two replicas. They add their counts to a 10 s window in advarapaladatabase on the CloudNativePG cluster, so a visitor cannot double their allowance by landing on both.
The Figure
The figure is two tollbooths with a hopper between them. Each booth is a dvarapala replica. The gauge on its front is its own token bucket for one visitor: 12 bars of 5 tokens, refilled at 6 a second. The hopper is the shared count in Postgres. Every second (the lamp on top blinks), both booths add what they let through. The hopper gauge fills toward 120 for the current 10 second window, and the pips above it count the seconds. A barrier lifts for each request that goes through. When a booth’s bucket is empty, or the hopper passes 120, the barrier stays down and the lamp on the booth’s roof lights. That is a 429. Click the figure, or press 1 to 5, to change the flow from 0 to 30 requests per second. Built with the anatomy skill.
The Situation
- Both apps are reached through a Cloudflare Tunnel, then the Cilium
public-gatewayin the k0s cluster (Part 9). - The old Cloudflare rule: 60 requests per 10 s per IP and Cloudflare data
center, then a 10 s block, on homie-demo
/api/and/auth/and ipcalc/api/. On the free plan, that is the whole feature: one rule, a 10 s period, a 10 s block. I had to add every public app to that one expression. - dvarapala 0.6.2, from
ghcr.io/franzramadhan/dvarapala. Version 0.6.1 was the first with a container image, and 0.6.2 is the first image with the Postgres store compiled in (more on that below). - CloudNativePG cluster
pg, two instances on NVMe data disks (Part 8), passwords from OpenBao through External Secrets (Part 5).
Setup
dvarapala does one upstream, the gateway does the routing
dvarapala proxies to exactly one upstream and has no host or path matching.
It rate limits everything it receives. That decided the layout. Each app gets
its own dvarapala Deployment and Service, and the HTTPRoute picks which paths
go through it. Static pages and assets skip it.
flowchart LR V[Visitor] --> CF[Cloudflare edge] --> T[Tunnel] --> GW[Cilium public-gateway] GW -->|/api/, /auth/| D[dvarapala<br/>2 replicas] GW -->|everything else| H[homie-demo] D --> H D -. counts every 1 s .-> PG[(CNPG pg<br/>db dvarapala)]
rules:
- matches:
- path: {type: PathPrefix, value: /api/}
- path: {type: PathPrefix, value: /auth/}
backendRefs:
- name: dvarapala
port: 80
- backendRefs:
- name: homie-demo
port: 80
The whole configuration is environment variables, so there is no ConfigMap. Lists are JSON:
env:
- name: DVARAPALA__UPSTREAM
value: http://homie-demo.homie-demo.svc.cluster.local
- name: DVARAPALA__TRUSTED_PROXIES
value: '["10.0.0.0/8", "192.168.101.0/24"]'
- name: DVARAPALA__RATE_LIMIT__RATE
value: "6"
- name: DVARAPALA__RATE_LIMIT__BURST
value: "60"
# every peer is a gateway envoy: no per-peer connection cap
- name: DVARAPALA__L4__EXEMPT
value: '["10.0.0.0/8", "192.168.101.0/24"]'
The probes are tcpSocket, not HTTP. dvarapala proxies every HTTP request,
so an HTTP probe tests the app behind it, not dvarapala. The pod runs as 10001
from a FROM scratch image with a read-only root filesystem. It requests
16 Mi and has 64 Mi as the limit. The CiliumNetworkPolicy lets it receive from
the gateway. It can only talk to DNS, the app pods and, on homie-demo,
Postgres on 5432.
Who gets counted
A rate limiter behind three proxies is only useful if it counts the visitor,
not the last proxy. The request reaches dvarapala with an X-Forwarded-For
header. Cloudflare started it, and every hop after it appended to it.
dvarapala takes the rightmost address that is not in trusted_proxies.
Pods are in 10.0.0.0/8 and the nodes in 192.168.101.0/24, so that address
is the one Cloudflare saw. If a client writes an address into the header
itself, it is further left, and dvarapala ignores it.
l4.exempt matters for a related reason. dvarapala also caps open
connections per peer (max_per_client = 256 by default), and here every TCP
peer is a gateway envoy. Without the exemption, the whole internet shares
256 connections.
I checked the keying through the real path instead of trusting the config. A
pod inside the cluster sent 120 parallel requests to the public gateway with
Host: ipcalc.franzramadhan.dev and a made-up X-Forwarded-For: 198.51.100.7:
# as 198.51.100.7
63 x 404 (ipcalc's own answer, the test skipped Cloudflare's headers)
57 x 429
# one request as 198.51.100.8 right after
1 x 404
The Cilium envoy kept the header and appended its own trusted hop. So .7
had its own bucket, and .8 had a fresh one.
Shared counts in CloudNativePG
One replica is a single point of failure in front of the app. Two replicas
that each keep their own memory give a visitor two buckets, and that doubles
the limit. dvarapala solves that with a store (Redis or Postgres). Nothing on
the request path waits for the store. Each replica keeps its local bucket.
Every second, in the background, it adds the requests it allowed per client
to a shared count. The count is for a 10 s window aligned to the clock. When
a client’s count for the window passes burst + rate x 10 s (60 + 60 = 120),
every replica throttles that client until the window ends.
I build the release binaries and the first image without the store features,
because most people run one process. So the image now gets its own static musl
build with --features redis,postgres (5.0 MB), released as 0.6.2. The
tarballs and the Homebrew formula did not change.
On the database side, I use the same pattern as the other apps on pg. The
Cluster spec manages the role, with its password from an ExternalSecret. A
Database resource creates the database:
# Cluster pg, spec.managed.roles
- name: dvarapala
ensure: present
login: true
passwordSecret:
name: pg-dvarapala
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: dvarapala
spec:
cluster:
name: pg
name: dvarapala
owner: dvarapala
dvarapala creates its own tables at start, in a dvarapala schema, as
UNLOGGED. PostgreSQL does not replicate them to the standby and empties them
after a crash or failover. Counts that live for 10 s do not need to survive
either event.
The password is in OpenBao twice: at databases/dvarapala for CNPG and at
homie-demo/dvarapala for the app. The reason is the Kyverno policy. It only
lets the ExternalSecrets of each namespace read their own path. The app side
templates the whole connection string:
target:
name: dvarapala-store
template:
data:
url: "postgres://{{ .username }}:{{ .password }}@pg-rw.databases.svc:5432/dvarapala?sslmode=verify-full"
verify-full needs the CA. The image has no CA bundle at all (it is FROM scratch). So the lab root CA is a ConfigMap, mounted at /etc/lab-ca and set
as DVARAPALA__STORE__CA_FILE. The Deployment has 2 replicas, a topology
spread across nodes and a PodDisruptionBudget of minAvailable: 1.
The first rollout raced itself
Flux applied the role, the Database and the Deployment together. The Database reconciled before CNPG created the role:
role "dvarapala" does not exist (SQLSTATE 42704)
and the dvarapala pods started before the database existed:
msg="store unavailable" ... database "dvarapala" does not exist (3D000)
Neither needed a hand. CNPG retried the Database when the role was there.
dvarapala kept serving on its local buckets (it fails open when the store is
gone). Then it reconnected and logged store schema migrated version=1 and
then store recovered. Both pods reported dvarapala_store_up 1 and
dvarapala_store_replicas 2 within about 15 s.
ipcalc stays on one dvarapala replica with no store. It is a stateless calculator, and a single replica has exact counts. A store only adds a dependency there.
What Changed for Visitors
The steady rate is the same. The shape around it is not.
| Cloudflare rule | dvarapala | |
|---|---|---|
| Counting | 60 per fixed 10 s period, per IP and Cloudflare data center | token bucket per visitor IP: 60 to start, 6 back per second |
| Most requests from idle in the first 10 s | 60 | about 120 (60 burst plus 60 refilled) |
| Sustained flood | 60 per 10 s | 6 per second, the same 60 per 10 s |
| Over the limit | 429, retry-after: 10, blocked for 10 s | 429, retry-after: 1, the next token is 1/6 s away |
| Where the 429 comes from | Cloudflare’s edge | inside the cluster, after the tunnel and the gateway |
| Two replicas | not applicable | one shared allowance of 120 per 10 s window |
A well-behaved client that retries after retry-after now waits 1 s instead
of 10. A client that keeps sending at full speed still gets 6 requests a
second and nothing more.
Here are the numbers from the real thing. While the Cloudflare rule still
existed, I sent 200 parallel requests from my laptop to ipcalc’s /api/.
They gave 93 x 200 and 104 x 429 from dvarapala (retry-after: 1). Only 3
got the Cloudflare retry-after: 10, so dvarapala already did nearly all the
limiting. After I deleted the rule, I sent the same burst against homie-demo.
It gave 54 x 429, all retry-after: 1. homie-demo itself answered the other
146 (a 404, because the test path does not exist). Not one came from
Cloudflare.
The metrics showed the shared count at work:
# pod 1
dvarapala_requests_total{verdict="allow",protocol="http1"} 75
dvarapala_requests_total{verdict="throttle",protocol="http1"} 26
# pod 2
dvarapala_requests_total{verdict="allow",protocol="http1"} 72
dvarapala_requests_total{verdict="throttle",protocol="http1"} 18
dvarapala_requests_total{verdict="throttle",protocol="http1",scope="cluster"} 10
Pod 2 turned away those 10 requests with scope="cluster" while its own
bucket still had tokens. The window count from both pods was over 120.
Sharing is approximate by design. A count reaches the other replica within two
sync rounds. So the worst-case overshoot per window is replicas x rate x 2 x (sync interval + store timeout) = 2 x 6 x 2 x 1.5 s = 36 requests. For a
demo app, that is a fair price. In return, the database is never on the
request path.
What Cloudflare still does
Cloudflare still sits in front: TLS, the tunnel, its DDoS protection for floods at the network edge, and the offline page at night. Only the per-IP limit on the API paths moved into the cluster. There is a cost. An over-limit request now travels through the tunnel and the gateway before dvarapala refuses it. Before, the edge dropped it for free. At demo traffic, that cost is nothing. In a real attack it matters, and the edge protection is the layer for that.
Known Gap: No Throttle Logs, Metrics Nobody Reads
dvarapala logs nothing when it throttles a request. It only counts, and its
metrics listen on 127.0.0.1:9090 inside the pod. Every number above came from
a kubectl port-forward. Today nothing scrapes those metrics, and no alert
fires on a burst of 429s. So I only notice an attack if I look. To get them
into VictoriaMetrics with an alert on throttle spikes, I need a separate
change.
I closed this gap two days later. See the update below.
Update 2026-10-30: the gap is closed
dvarapala now reports into the same VictoriaMetrics, VictoriaLogs and Grafana
as everything else (Part 28),
with its own dashboard and alerts. By now both apps run dvarapala 0.15.0
with a dvarapala.toml ConfigMap instead of the environment variables above.
ipcalc runs one replica, and homie-demo runs two.
The logs were already there, just unreadable
The otel-agent DaemonSet collects the stdout of every pod. So the dvarapala lines were in VictoriaLogs from day one (about 57 an hour from homie-demo, 5 from ipcalc). Nobody was able to filter them. dvarapala writes coloured terminal output, and VictoriaLogs stored every line with its escape codes:
\u001b[2m2026-10-30T08:12:00.693808Z\u001b[0m \u001b[32m INFO\u001b[0m \u001b[2mdvarapala::admin::serve\u001b[0m ...
dvarapala logs through tracing_subscriber, which honours the NO_COLOR
convention (dvarapala’s own tests set it). So the fix is one environment
variable on the container, with no code change:
env:
# plain key=value log lines in VictoriaLogs, no ANSI colour codes
- name: NO_COLOR
value: "1"
2026-10-30T08:28:40.388725Z INFO dvarapala: proxy listening listen=0.0.0.0:8080 ...
Now a LogsQL word filter such as k8s.container.name:dvarapala ERROR matches
what it says.
Metrics off the loopback
The metrics listener defaults to 127.0.0.1:9090. To move it to all
interfaces, I added one TOML section and a container port named metrics:
# Prometheus endpoint, scraped by otel-cluster (job dvarapala); the
# network policy lets only that collector reach it
[metrics]
listen = "0.0.0.0:9090"
A listener on the pod IP is not public. The HTTPRoute only points at port 80
of the dvarapala Service. The CiliumNetworkPolicy gets one more ingress rule
that lets only the cluster collector in:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
app.kubernetes.io/instance: otel-cluster
toPorts:
- ports: [{port: "9090", protocol: TCP}]
The collector scrapes it like the other apps, by pod label and port name:
- job_name: dvarapala
scrape_interval: 30s
kubernetes_sd_configs:
- role: pod
namespaces:
names: [homie-demo, ipcalc]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name, __meta_kubernetes_pod_container_port_name]
regex: dvarapala;metrics
action: keep
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
A minute after the pods rolled, VictoriaMetrics had up = 1 for all three
pods and dvarapala_store_up = 1 on both homie-demo replicas.
A dashboard per question

The dashboard has an instance switch (homie-demo, ipcalc or both), and each
row answers one question:
| Row | Question |
|---|---|
| Overview | Is anyone being refused, and by what? Verdicts per second, refusals by scope (empty for the local bucket, cluster for the shared Postgres count, block for a manual block), per route, upstream errors, visitors in the limiter table, detection state |
| Challenge | Do visitors pass the proof of work on the homie-demo login, or fail it? |
| WAF | What would the Coraza sidecar block in enforce? Verdicts, the top CRS rule IDs over an hour, inspect latency, agent errors |
| Shared store | Is the Postgres sync keeping up? Sync p95, replicas seen, errors |
| Console and admin | Who logged in to the dvarapala console, and did a config reload fail? |
| Resources | CPU and memory of dvarapala and the WAF sidecar against their limits |
| Logs | dvarapala and coraza-spoa lines from VictoriaLogs, with a LogsQL filter box |
I wanted the WAF row most. Both apps run the WAF in observe. The plan is to
switch to enforce after some days without false positives. That decision now
has a panel instead of a grep for waf match.
Seven alerts
They go to the same Telegram and email contact point as the rest (Part 32):
| Alert | Fires when |
|---|---|
| upstream errors | more than 0.1 failed upstream requests a second for 5 min: visitors get 502 |
| store down | dvarapala_store_up is 0 for 5 min: the homie-demo replicas fall back to their own buckets, so a visitor gets about twice the limit |
| WAF agent errors | any coraza-spoa failure in 10 min: requests pass with verdict error and nothing is inspected |
| attack detected | adaptive detection is in Attack for 2 min (it is in observe, so this tells me, it does not block) |
| config reload failed | a reload was rejected: the old config keeps running, silently |
| console logins refused | 3 or more denied or failed console logins in 15 min |
| error logs | more than 5 ERROR lines in 10 min, from VictoriaLogs |
There is no “dvarapala down” alert. The existing Metrics target down rule
is max by (service_name, pod)(up) over every scrape job. So it already fires
for a dvarapala pod that stops answering. A second rule only sends the same
page twice.
The thresholds are a first guess at demo-sized traffic. I will check them against a few days of data.
Two Grafana surprises
The upstream errors summary used {{ humanize $values.A.Value }}. On the very
first evaluation, before the series existed, Grafana logged:
error calling humanize: can't convert <nil> to float
humanize fails on a missing value. printf "%.2f" prints it without an
error. The other trap: the threshold expression has gt, lt and ranges but
no eq. So I wrote “detection state is 2 (Attack)” as gt 1.5 on
max(dvarapala_detection_state).
Why XDP stays off
dvarapala can also run an XDP program that drops blocked sources and packet floods in the NIC driver (the dvarapala post has the details). I looked at turning it on here and left it off. The reason has nothing to do with dvarapala:
flowchart LR V[Visitor<br/>real IP] --> CF[Cloudflare edge] CF --> T[cloudflared<br/>outbound tunnel] T --> GW[gateway envoy<br/>10.0.0.0/8] GW -->|packet source = envoy| D[dvarapala pod]
- No visitor packet ever reaches the lab. The tunnel is an outbound
connection from cloudflared. So the packets that arrive at a dvarapala pod
come from a gateway envoy, a pod or node address. The visitor’s IP only
exists in
X-Forwarded-For, which is L7. XDP keys its block list and packet rates on the packet’s source prefix. So it sees a handful of envoys and nobody else. - Enforce points the gun at the gateway. Those addresses are already
in
l4.exemptfor the same reason. An XDP block or a packet rate on them cuts off every visitor at once. - It needs a different build and more privilege. The published image is
built with
--features redis,postgres, notxdp. The pod also needsCAP_BPFandCAP_NET_ADMIN, and the restricted pod security profile does not allow them.
XDP belongs on a host that receives the internet’s packets directly. Here that host belongs to Cloudflare, and its DDoS protection is the layer for packet floods.
Lessons
- Compare the whole curve, not the headline number. “60 per 10 s” fits both limiters, but it describes two different experiences. One is a fixed window with a 10 s block. The other is a bucket that allows a bigger first burst and then refuses only for a second at a time.
- Let the gateway route and the limiter limit. The single upstream in dvarapala looked like a limitation until the HTTPRoute did the path selection. Each piece stays simple.
- Test who the limiter thinks the client is, through the real path. If the trusted proxy list is one hop off, the limiter counts the gateway instead of the visitor. Then every visitor shares one bucket.
- Shared state can be approximate. Background sync, fail open and a known overshoot of 36 are better than a database call on every request.
- The Postgres you already run is enough for counters. It takes a managed
role, a
Databaseresource and unlogged tables. There is no Redis to add and operate. - Check what the collector already has before adding a pipeline. The logs
were in VictoriaLogs all along. The fix was
NO_COLOR=1, not a new shipper. - A packet-level defence needs the attacker’s packets. Behind a tunnel and a gateway, the source address on the wire is your own proxy. So XDP has nothing to tell apart. Put it where the internet’s packets land.
Series
This post is part of Homelab Rebuild:
- Part 1: Upgrading Proxmox VE 8 to 9
- Part 2: Starting Over When the IaC Is Broken
- Part 3: When Tailscale Breaks Your Cilium Gateway
- Part 4: Living on the Tailnet
- Part 5: Secrets Without Secrets in Git
- Part 6: Certificates, DNS and a Private CA
- Part 7: Self-Hosted CI and Getting Back to IaC
- Part 8: Stateful Workloads on Unequal Hardware
- Part 9: Going Public Safely with Cloudflare Tunnel
- Part 10: When the Hypervisor Went Quiet
- Part 11: Hardening the Cluster in Five Layers
- Part 12: Observability with Coroot, and Losing pve-01
- Part 13: The Host That Kept Losing Power
- Part 14: Observability, Take Two with OpenTelemetry
- Part 15: Requests and Limits for Everything
- Part 16: Handing the Cluster to FluxCD
- Part 17: Making the Guardrails Earn Their Keep
- Part 18: When the API Server Outgrew Its VM
- Part 19: Logging In to OpenBao with Cloudflare Access and Google
- Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
- Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
- Part 22: Turning Four Days of Fixes into AI Runbooks
- Part 23: microVM Sandboxes on Proxmox
- Part 24: Pi-hole Without a Home Host
- Part 25: It Was the Router
- Part 26: TLS on the Front, Plaintext Behind It
- Part 27: One Bad Power Adapter, Two Different Deaths
- Part 28: Observability, Take Three: Making It Worth Looking At
- Part 29: One Meter for Every Model, LiteLLM on Workers AI
- Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
- Part 31: First Backups, Postgres, OpenBao and etcd to R2
- Part 32: Alerts That Reach Me, Even When the Lab Is Dead
- Part 33: The Firewall Held, the Tailnet Did Not
- Part 34: From Watching to Blocking, and a Record of Who Did What
- Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
- Part 36: The Fans Were Loud, the CPUs Were Idle
- Part 37: A Power Trip, a Search Domain and a Wildcard
- Part 38: An Autonomous AI Pentester in a Sealed Box
- Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed
- Part 40: Taking Rate Limiting Back from Cloudflare (this post)
References
- GitHub: franzramadhan/dvarapala
- Cloudflare: Rate limiting rules
- CloudNativePG: Declarative database management
- CloudNativePG: Database role management
- Gateway API: HTTPRoute
- PostgreSQL: CREATE TABLE (UNLOGGED)
- GitHub: wheresryan22/anatomy
- NO_COLOR: Disabling ANSI color output by default
- Grafana: Template annotations and labels
- Cilium: Layer 3 policies (endpoints based)
