dvarapala: An Anti-DDoS Gateway in Rust, from L7 Down to XDP

Intro

Every public service I run sits behind something that is supposed to stop a flood: Cloudflare’s free plan, a rate limit rule, a WAF. Each covers one layer, and each stops where its plan stops. A SYN flood is a different problem from a slowloris client, which is a different problem from a thousand clients that each stay just under the rate limit. And when the service calls out to third-party APIs, the flood can be me: my own retries hitting a provider until it blocks my key.

So I wrote dvarapala, a gateway that handles all of those in one small binary. It is named after the Dvarapala, the guardian statues at Javanese temple gates. Docs live at dvarapala.franzramadhan.dev.

TL;DR: dvarapala is a Rust reverse proxy, forward proxy and embeddable library with protection from L4 to L7: connection limits, TLS, HTTP/2 and HTTP/3 limits, a token bucket per client, adaptive flood detection, and egress rate shaping toward third-party APIs. v0.6.0 adds shared state across replicas through Redis or PostgreSQL and an optional XDP program that drops packet floods in the NIC driver. It went from the first RFC to v0.6.0 in two days, 11 RFCs, 241 tests at 89% line coverage. The idle binary uses about 6 MB of memory and is 4.4 MB on disk. Source: github.com/franzramadhan/dvarapala (MIT).

The Figure

The figure is dvarapala as a temple gatehouse: requests walk the causeway toward the shrine (the upstream), through each stage in the order the binary checks them. Drag the attack slider and switch between observe and enforce to see where traffic stops, and watch a blocked source travel down the conduit to the sister gates (the cluster). I built it with the anatomy skill (see References).

What It Does

Three ways to run it:

And the layers, each optional, each its own RFC:

Installing it is one line on macOS or Linux. The script picks the release for the platform and checks it against the release’s SHA256SUMS before installing:

brew install franzramadhan/tap/dvarapala
# or
curl -fsSL https://dvarapala.franzramadhan.dev/install.sh | sh

The smallest config is two lines:

listen = "0.0.0.0:8080"
upstream = "http://127.0.0.1:8081"

Everything else has a default: 20 requests per second per client (burst 40), detection in observe mode, Prometheus metrics on 127.0.0.1:9090. Detection starts in observe on purpose: it logs and counts what it would do, and changes no verdict until I have watched it against real traffic and set mode = "enforce".

The Situation

• dvarapala 0.6.0, Rust 1.99, tokio, hyper, tower, rustls, quinn and h3 for HTTP/3, redis and tokio-postgres for shared state, aya for XDP.
• One crate per domain: engine (verdicts, no I/O, no async runtime), l4, l7, proxy, store, xdp, ffi and the binary. The engine has no I/O so the proxy and the SDKs share the same verdict logic.
• CI fails if the idle binary’s RSS goes over 10 MB, if HTTP/3 adds more than 2 MB to the binary, or if XDP adds more than 1 MB. Today: about 6 MB idle, HTTP/3 adds 0.9 MB, XDP adds 0.4 MB.
• Every parser that reads attacker input has a cargo fuzz target, and CI fuzzes each for 60 seconds on every PR.

How a Packet Gets Through

flowchart LR
  P[packet] --> X{XDP<br/>block list, SYN rate}
  X -->|drop| D1[dropped in driver]
  X --> L4{L4<br/>conn caps, accept rate}
  L4 -->|close| D2[closed]
  L4 --> T[TLS or QUIC]
  T --> R{rate limit<br/>token bucket}
  R -->|429| D3[Retry-After]
  R --> DT{detection}
  DT -->|enforce: 403| D4[dropped]
  DT --> U[upstream]

Each stage is cheaper than the one after it, so the expensive work only happens for traffic that survived the cheap checks. The XDP program runs before the kernel even builds a socket buffer. L4 closes a refused connection before any TLS or HTTP is read. The rate limit answers 429 without contacting the upstream.

The stages also feed each other. In enforce mode, a client the L4 accept rate refuses, or a heavy hitter that detection drops, gets a dynamic block in the XDP program for 10 minutes. Both have completed a TCP handshake, so the source address is real: a spoofed SYN flood cannot trick dvarapala into blocking someone else.

How It Was Built

Every feature starts as an RFC in rfc/, from a template with a “Test cases” table. I accept it before any code is written. Then the implementation lands as a stack of PRs:

flowchart LR
  RFC[RFC accepted] --> T[tests layer<br/>cases ignored over stubs]
  T --> E[engine layer] --> M[middle layers<br/>l4, l7, store, xdp]
  M --> C[cli layer<br/>config, wiring, docs]
  C --> G{all checks green}
  G --> S[one squash merge<br/>per layer]

The bottom layer adds every test from the RFC’s table, marked #[ignore = "rfc NNNN"] over todo!() stubs, so the tests exist and compile before the code does. Each layer above makes some of them pass. The stack merges in one step once every layer is green. A release PR then marks the RFCs implemented and has to update the README, the config reference and the docs site: a CI job fails the release if an implemented RFC is not mentioned in all three.

Issue 1: Replicas Lost Sight of Each Other in CI

Symptom

The PostgreSQL cluster tests passed on my laptop and failed in CI, at random: a client spread over three replicas was allowed more than the cluster limit, and the heavy hitter one replica dropped was not dropped on the others.

Cause

Each replica writes a heartbeat every sync round, and the others count it as live while the heartbeat is fresh. I had set the expiry to three sync intervals. Under load in CI, one round took over 200 ms, so a heartbeat could expire between two rounds. A replica that thinks it is alone does not split the budget, so the cluster allowed more than the limit.

Fix

The expiry is now 3 x sync_interval + timeout: a heartbeat outlives the slowest round that is still allowed to succeed. Before changing anything I first made the tests print the replica count and the round time, which turned “flaky” into one clear number.

Issue 2: Advisory Locks and a Connection Pooler

Symptom

My first version took a session advisory lock so that only one replica at a time sweeps expired rows. That works against PostgreSQL directly. I run CloudNativePG at home, and its Pooler is PgBouncer in transaction mode, where the lock would not hold.

Cause

Session advisory locks belong to a server connection. In transaction mode the pooler hands a different server connection to each transaction, so a lock taken in one transaction says nothing about the next.

Fix

Transaction-scoped locks (pg_advisory_xact_lock, pg_try_advisory_xact_lock), released at commit on the same connection they were taken on. Keys are sorted before the upsert so two replicas never deadlock on each other’s rows, and the sweep deletes at most 10 000 rows per table per round so it never holds a lock for long.

Issue 3: The XDP Program Has to Be Reproducible

Symptom

The XDP program is a separate eBPF object, built with a nightly Rust and bpf-linker. Building it on every cargo build would make everyone install that toolchain, but checking in a binary nobody can rebuild is worse.

Cause

An eBPF object depends on the exact compiler and LLVM version. A different nightly gives different bytes, and possibly a program the kernel verifier rejects.

Fix

The object is checked in, so a normal build needs no extra toolchain. The program lives in its own workspace (crates/xdp-ebpf) with the nightly pinned in rust-toolchain.toml, and CI downloads bpf-linker 0.12.0, checks its sha256, rebuilds the object and fails if it differs from the checked-in one by a single byte. Another CI step loads the program into the runner’s kernel as root and runs the RFC’s test cases through BPF_PROG_TEST_RUN: the block list, the SYN rate, observe mode, expiring blocks, and that a killed process leaves no program attached.

Lessons

1. Cheap checks first, and let them feed each other. XDP drops what L4 and L7 already proved is hostile, so the expensive layers only see traffic worth their time.
2. Only block what completed a handshake. A dynamic block list fed by spoofable packets is a tool for attackers to block your real users.
3. Write the test cases in the RFC. A table of cases written before the code is the cheapest review I know, and the bottom layer of each stack turns it into tests that must pass before the stack can merge.
4. Make the budget a test. Memory, binary size and fuzzing all fail CI when they regress, so “small” stays true without anyone remembering to check.

References

• GitHub: franzramadhan/dvarapala
• dvarapala: Documentation
• Linux kernel: BPF documentation
• GitHub: aya-rs/aya
• PostgreSQL: Advisory locks
• PgBouncer: Features by pool mode
• GitHub: wheresryan22/anatomy