Intro
dvarapala got an optional [ml]
layer in RFC 0002, the second one. An XGBoost model that you bring scores
every request that the rate limiter allows, per client. A score above a
threshold turns the answer into 429. For a long time, nobody was able to use it
outside a source build, because no published binary or image had the ml
feature. v0.19.1 fixed that. The container image now has every feature,
so the gateways that the operator runs can load a model.
That left the harder half. A model needs labeled rows. The gateway computes the ten dvarapala features itself, so no public dataset has them. The LLM label sampler from RFC 0023 already writes those feature vectors next to each label. This post is the full loop on one real deployment: synthetic traffic, labels, training, a model mounted into the gateway, and what the scores look like.
TL;DR: four synthetic client profiles (benign, scraper, scanner, flood), 30 minutes, 236 labeled rows from
dvarapala.llm_labels. Ground truth came from a tag in the path, not from the LLM. The LLM agreed with it on 93.2% of rows for “attack or not”, but it called almost every attacker a scanner. XGBoost 3.1.1 reached 100% on the held-out rows with one feature only: all 30 trees askrate_ewma < 1.054?. Benign clients score 0.026, everything else 0.993. The model runs withthreshold = 1.0(watch only). Anml_bandtrigger sends every client with a score of 0.5 or more to the LLM for a second opinion. The split really means “never sent two requests in one second”, and a real visitor breaks that with a double click. A test run atthreshold = 0.9throttled 94% of the scraper and no benign client. Then I set the threshold back to 1.0.
The Figure
The figure is a punch-card tabulator. Each card is one client’s request.
Only its rate_ewma row lies under the 30 pins, one per tree. Pick a client
and watch the pins. On a benign card, they all drop through the punched row
and the dial settles at 0.026. A scraper, scanner or flood card stops every
pin, and the dial climbs to 0.993. Set the threshold to 0.9 or 0.5, and
those cards go to the red 429 tray. At 1.0, the watch lamp is lit. Every
card goes to the pass tray, and the tabulator only records the score.
The figure is drawn with the anatomy skill.
The Situation
- dvarapala 0.19.1, operator and gateways, on the lab’s Kubernetes cluster.
- The gateway in front of ipcalc’s JSON API (
/api/v1/) has three parts. A token bucket of 60 per client, refilled at 6 per second. A Coraza WAF sidecar in observe mode. The LLM label sampler ([ml.llm]), which writes rows to Postgres (persist = true). - Python 3.13 for the load generator, XGBoost 3.1.1 for training, both in throwaway containers.
What dvarapala gives a model, per client (detection.window is 1 s):
| Feature | Meaning |
|---|---|
window_requests | Requests in the current window |
rate_ewma | Average requests per window |
iat_mean, iat_var | Mean and variance of the seconds between requests |
size_mean | Average body size |
no_ua_share | Share of requests without User-Agent |
distinct_paths | Distinct paths in the window, at most 8 |
status_4xx_share | Share of 4xx responses |
global_rate_z, source_entropy | Instance-wide detection state |
The averages move 10% toward each new value. That matters later.
Setup
The pipeline
flowchart LR L[load generator] --> G[dvarapala gateway] G --> A[ipcalc API] G -- sampled, redacted --> M[LLM] M -- label --> P[(llm_labels)] G -- features --> P P --> T[train.py] T --> C[ConfigMap model.json] C --> G
The gateway samples requests and redacts them (query values become _, keys
stay). It asks the LLM for one label. Then it stores the label with the
client’s feature vector at that moment. Training reads those rows. The model
goes back to the same gateway as a mounted file.
Synthetic traffic with ground truth
A Python pod inside the cluster talked to the gateway’s Service directly.
It set X-Forwarded-For (the gateway trusts the cluster network as a proxy)
to addresses from 198.51.100.0/24, the RFC 5737 documentation range. Each
profile had eight clients:
| Profile | Shape | User agent |
|---|---|---|
| benign | one valid API call every 3 to 15 s | Firefox |
| scraper | one valid call per second, steady | python-requests/2.32.3 |
| scanner | probe paths (/.env, /.git/config, wp-login.php), SQLi and XSS in queries, every 0.3 to 1.5 s | none |
| flood | bursts of 120 over 6 connections, every 5 to 10 s | curl/8.11.1 |
The trick for ground truth: each profile adds its own query key, t0 to
t3. Redaction drops the value but keeps the key. So a stored path like
/api/v1/calc?q=_&t1=_ tells which profile sent it. The LLM never knows.
The benign and scraper clients have no WAF match and stay under the rate
limit. To get rows for them, I added a temporary route for /api/ and a
route trigger with per_client = "5m". So the gateway sampled every
client every five minutes.
def scraper(ip): # steady machine pace under the rate limit, script UA
while time.time() < END:
get(ip, valid("t1"), "python-requests/2.32.3")
time.sleep(1.0 + random.uniform(-0.05, 0.05))
After 30 minutes:
| Profile | Responses |
|---|---|
| benign | 1604 x 200 |
| scraper | 13724 x 200 |
| scanner | 12111 x 404, 3099 x 400 |
| flood | 86022 x 200, 97096 x 429 |
The scraper never saw a 429: one request per second is well under 6. The
rate limiter cannot catch that client, and I wanted the model for it.
Labels: good at “bad”, bad at “which bad”
psql exported 240 rows. Four had null features, and I dropped them. That
left 236. Here is the LLM’s label against the tag:
profile llm label rows
benign benign 35
benign scanner 13
flood benign 2
flood scanner 42
scanner scanner 96
scraper benign 1
scraper scanner 45
scraper scraper 2
As a binary question (benign or not), the LLM agreed with the ground truth
on 93.2% of rows. As a type, it was mostly wrong. A steady python-requests
client and a curl flood both came back as scanner. The 13 benign rows
labeled scanner came from my first run. In that run, a wrong endpoint
answered 404 to everyone. status_4xx_share moves only 10% per request.
So it was still high minutes later, and the LLM saw it.
So I trained on the tag, not the LLM label. Real traffic has no tag, so the LLM label (filtered by confidence) is what you have. This run says that it is usable for the binary target and not for anything finer.
Training
booster = xgb.train(
{"objective": "binary:logistic", "max_depth": 3, "eta": 0.3},
dtrain, num_boost_round=30)
75% train, 25% held out:
held-out scores by profile (min / median / max):
benign n= 10 0.026 / 0.026 / 0.026
scraper n= 9 0.993 / 0.993 / 0.993
scanner n= 27 0.993 / 0.993 / 0.993
flood n= 13 0.993 / 0.993 / 0.993
held-out accuracy at 0.5: 100.0%
gain by feature: {'rate_ewma': 24.8}
One feature. model.json shows why the scores are two numbers. Each of the
30 trees has a single split, rate_ewma < 1.054, with one leaf for yes and
one for no (tree 0: -1.209 and +0.366). Every benign client lands on the
same 30 leaves, and every other client lands on the other 30. The sum goes
through the logistic function to 0.026 or 0.993.
That is not a bug in the training. It comes from the data. Here is
rate_ewma per profile in the training rows:
| Profile | min | median | max |
|---|---|---|---|
| benign | 1.000 | 1.000 | 1.000 |
| scraper | 1.054 | 1.210 | 1.527 |
| scanner | 1.266 | 1.485 | 1.824 |
| flood | 23.107 | 31.051 | 40.243 |
Every benign row is exactly 1.0. The average only moves when a window has a request. So a client that never sends two requests in the same second stays at 1.0 forever. The scraper’s 1 s sleep has 50 ms of jitter. Now and then, two of its requests land in one window, and the average creeps up. XGBoost found the cheapest separation: “did this client ever send two requests within one second”.
Real visitors are messier. A person can click twice within a second, or a page can make two API calls for one action. Either one moves the average from 1.0 to 1.1. The client then scores 0.993 for its next six requests, until the 10% steps bring it back under 1.054. In enforce mode, a model trained on my synthetic benign profile throttles real people.
XGBoost 3 model format
XGBoost 3 saves base_score as a string that holds an array,
"[7.8531075E-1]". Older versions wrote a plain number. Before I deployed,
I made sure that 0.19.1 reads it. A local docker run of the image with the
model in [ml] started, and six requests gave
dvarapala_ml_score_count 6.
The model in the gateway
The model is a file in a ConfigMap, mounted through the operator’s
gateway.mounts:
configMapGenerator:
- name: dvarapala-model-v1
files:
- model.json=dvarapala-model.json
[ml]
model = "/etc/dvarapala-mounts/model/model.json"
threshold = 1.0
[[ml.llm.triggers]]
on = "ml_band"
band = [0.5, 1.0]
The gateway reads the model once at start, so the ConfigMap name carries a version. A new model means a new name, a changed pod spec and a rolled pod.
threshold = 1.0 is the watch-only setting. A score is never above 1, so
the gateway throttles nothing, and dvarapala_ml_score records every score.
The ml_band trigger sends clients with a score between 0.5 and 1.0 to the
LLM. So the model’s suspects get a second opinion that I can read in the
console. I removed the temporary route and its trigger in the same change.
What the Scores Look Like
A five minute rerun of the same load against the gateway with the model:
dvarapala_ml_score{quantile="0"} 0.026153777535046534
dvarapala_ml_score{quantile="0.5"} 0.9929251404657734
dvarapala_ml_score{quantile="0.99"} 0.9929251404657734
dvarapala_ml_score_count 19566
Two values, as trained. The ml_band trigger picked all 24 attacking
clients and none of the benign ones:
| Profile | ml_band samples | LLM label | Mean confidence |
|---|---|---|---|
| scraper | 8 | scanner | 0.42 |
| scanner | 8 | scanner | 0.82 |
| flood | 8 | scanner | 0.52 |
The LLM is least sure about the scraper. That makes sense: it sends valid requests with a plausible user agent, and only the pace is wrong.
Turning it on
To see the other half work, I set threshold = 0.9 for one more five
minute run of the same load. Then I put it back to 1.0:
| Profile | Watching (1.0) | Enforcing (0.9) |
|---|---|---|
| benign | 282 x 200 | 267 x 200 |
| scraper | 2285 x 200 | 141 x 200, 2154 x 429 |
| scanner | 2057 x 404, 492 x 400 | 30 x 404, 271 x 400, 2256 x 429 |
| flood | 14682 x 200, 18798 x 429 | 8 x 200, 33592 x 429 |
The token bucket never touched the scraper, but now it lost 94% of its
requests to 429. The 141 that got through, about 18 per client, are most
likely the requests before each client first landed two in one second. The
scanner’s probes stopped reaching the app. The flood lost the requests that
the bucket let through between bursts before. Benign clients never saw a
429.
So enforcement does what the configuration reference says. I still set it back to 1.0, because of what the table in the training section shows. The benign profile here never double clicks, and real people do.
Lessons
- The model learns whatever separates your data most cheaply. With synthetic profiles that differ mostly in pace, that is one rate threshold. Make the benign traffic as varied as the attacks before you trust more than that.
- Look at the feature values behind a split. “rate_ewma < 1.054” reads like “about one request a second”. The rows show that it means “never two requests in one second”. A real visitor breaks that with one double click.
- LLM labels are good enough for “attack or not”. They had 93.2% agreement on the binary target, and they were poor on the type. Train on the binary label and keep the confidence filter.
- Moving averages remember.
status_4xx_sharecarried a bad first run into labels minutes later. Restart the load, or wait, after a mistake. - Watch before you enforce.
threshold = 1.0plusml_bandgives the score distribution and a second opinion on every suspect. No visitor is affected.
References
- dvarapala: Protection layers, scoring model
- dvarapala: Configuration,
[ml]and[ml.llm] - dvarapala: RFC 0002, adaptive detection
- dvarapala: RFC 0023, LLM labels
- XGBoost: Introduction to Model IO
- IETF: RFC 5737, IPv4 Address Blocks Reserved for Documentation
- GitHub: wheresryan22/anatomy
