Back to Home
post-thumb

dvarapala: Training an XGBoost Model from Its Own LLM Labels

| November 01, 2026 | 11 min to read

Intro

dvarapala got an optional [ml] layer in RFC 0002, the second one. An XGBoost model that you bring scores every request that the rate limiter allows, per client. A score above a threshold turns the answer into 429. For a long time, nobody was able to use it outside a source build, because no published binary or image had the ml feature. v0.19.1 fixed that. The container image now has every feature, so the gateways that the operator runs can load a model.

That left the harder half. A model needs labeled rows. The gateway computes the ten dvarapala features itself, so no public dataset has them. The LLM label sampler from RFC 0023 already writes those feature vectors next to each label. This post is the full loop on one real deployment: synthetic traffic, labels, training, a model mounted into the gateway, and what the scores look like.

TL;DR: four synthetic client profiles (benign, scraper, scanner, flood), 30 minutes, 236 labeled rows from dvarapala.llm_labels. Ground truth came from a tag in the path, not from the LLM. The LLM agreed with it on 93.2% of rows for “attack or not”, but it called almost every attacker a scanner. XGBoost 3.1.1 reached 100% on the held-out rows with one feature only: all 30 trees ask rate_ewma < 1.054?. Benign clients score 0.026, everything else 0.993. The model runs with threshold = 1.0 (watch only). An ml_band trigger sends every client with a score of 0.5 or more to the LLM for a second opinion. The split really means “never sent two requests in one second”, and a real visitor breaks that with a double click. A test run at threshold = 0.9 throttled 94% of the scraper and no benign client. Then I set the threshold back to 1.0.

The Figure

0000004290.0000.0260.993
clientthreshold
Dial needle: the score, 0 at the bottom, 1 at the topThreshold pointer and the 429 traySpring pins: the 30 treesEach card is one client's request, punched with ten features, but only the rate_ewma row sits under the pins. Every one of the 30 trees asks rate_ewma < 1.054: a benign browser's row is punched, so all 30 pins drop through (yes, a negative leaf; tree 0 adds -1.209), while a scraper, scanner or flood card stops every pin (no, a positive leaf; tree 0 adds +0.366). The leaf sum plus a base goes through a logistic curve, the inner ring of the dial: benign reads 0.026, the rest 0.993. A card reading above the red pointer goes to the 429 tray. At threshold 1.0 the pointer sits at the top, the watch lamp is lit, nothing is rejected and the score only lands in dvarapala_ml_score.

The figure is a punch-card tabulator. Each card is one client’s request. Only its rate_ewma row lies under the 30 pins, one per tree. Pick a client and watch the pins. On a benign card, they all drop through the punched row and the dial settles at 0.026. A scraper, scanner or flood card stops every pin, and the dial climbs to 0.993. Set the threshold to 0.9 or 0.5, and those cards go to the red 429 tray. At 1.0, the watch lamp is lit. Every card goes to the pass tray, and the tabulator only records the score.

The figure is drawn with the anatomy skill.

The Situation

  • dvarapala 0.19.1, operator and gateways, on the lab’s Kubernetes cluster.
  • The gateway in front of ipcalc’s JSON API (/api/v1/) has three parts. A token bucket of 60 per client, refilled at 6 per second. A Coraza WAF sidecar in observe mode. The LLM label sampler ([ml.llm]), which writes rows to Postgres (persist = true).
  • Python 3.13 for the load generator, XGBoost 3.1.1 for training, both in throwaway containers.

What dvarapala gives a model, per client (detection.window is 1 s):

FeatureMeaning
window_requestsRequests in the current window
rate_ewmaAverage requests per window
iat_mean, iat_varMean and variance of the seconds between requests
size_meanAverage body size
no_ua_shareShare of requests without User-Agent
distinct_pathsDistinct paths in the window, at most 8
status_4xx_shareShare of 4xx responses
global_rate_z, source_entropyInstance-wide detection state

The averages move 10% toward each new value. That matters later.

Setup

The pipeline

flowchart LR
  L[load generator] --> G[dvarapala gateway]
  G --> A[ipcalc API]
  G -- sampled, redacted --> M[LLM]
  M -- label --> P[(llm_labels)]
  G -- features --> P
  P --> T[train.py]
  T --> C[ConfigMap model.json]
  C --> G

The gateway samples requests and redacts them (query values become _, keys stay). It asks the LLM for one label. Then it stores the label with the client’s feature vector at that moment. Training reads those rows. The model goes back to the same gateway as a mounted file.

Synthetic traffic with ground truth

A Python pod inside the cluster talked to the gateway’s Service directly. It set X-Forwarded-For (the gateway trusts the cluster network as a proxy) to addresses from 198.51.100.0/24, the RFC 5737 documentation range. Each profile had eight clients:

ProfileShapeUser agent
benignone valid API call every 3 to 15 sFirefox
scraperone valid call per second, steadypython-requests/2.32.3
scannerprobe paths (/.env, /.git/config, wp-login.php), SQLi and XSS in queries, every 0.3 to 1.5 snone
floodbursts of 120 over 6 connections, every 5 to 10 scurl/8.11.1

The trick for ground truth: each profile adds its own query key, t0 to t3. Redaction drops the value but keeps the key. So a stored path like /api/v1/calc?q=_&t1=_ tells which profile sent it. The LLM never knows.

The benign and scraper clients have no WAF match and stay under the rate limit. To get rows for them, I added a temporary route for /api/ and a route trigger with per_client = "5m". So the gateway sampled every client every five minutes.

def scraper(ip):  # steady machine pace under the rate limit, script UA
    while time.time() < END:
        get(ip, valid("t1"), "python-requests/2.32.3")
        time.sleep(1.0 + random.uniform(-0.05, 0.05))

After 30 minutes:

ProfileResponses
benign1604 x 200
scraper13724 x 200
scanner12111 x 404, 3099 x 400
flood86022 x 200, 97096 x 429

The scraper never saw a 429: one request per second is well under 6. The rate limiter cannot catch that client, and I wanted the model for it.

Labels: good at “bad”, bad at “which bad”

psql exported 240 rows. Four had null features, and I dropped them. That left 236. Here is the LLM’s label against the tag:

profile  llm label   rows
benign   benign        35
benign   scanner       13
flood    benign         2
flood    scanner       42
scanner  scanner       96
scraper  benign         1
scraper  scanner       45
scraper  scraper        2

As a binary question (benign or not), the LLM agreed with the ground truth on 93.2% of rows. As a type, it was mostly wrong. A steady python-requests client and a curl flood both came back as scanner. The 13 benign rows labeled scanner came from my first run. In that run, a wrong endpoint answered 404 to everyone. status_4xx_share moves only 10% per request. So it was still high minutes later, and the LLM saw it.

So I trained on the tag, not the LLM label. Real traffic has no tag, so the LLM label (filtered by confidence) is what you have. This run says that it is usable for the binary target and not for anything finer.

Training

booster = xgb.train(
    {"objective": "binary:logistic", "max_depth": 3, "eta": 0.3},
    dtrain, num_boost_round=30)

75% train, 25% held out:

held-out scores by profile (min / median / max):
  benign   n= 10  0.026 / 0.026 / 0.026
  scraper  n=  9  0.993 / 0.993 / 0.993
  scanner  n= 27  0.993 / 0.993 / 0.993
  flood    n= 13  0.993 / 0.993 / 0.993
held-out accuracy at 0.5: 100.0%
gain by feature: {'rate_ewma': 24.8}

One feature. model.json shows why the scores are two numbers. Each of the 30 trees has a single split, rate_ewma < 1.054, with one leaf for yes and one for no (tree 0: -1.209 and +0.366). Every benign client lands on the same 30 leaves, and every other client lands on the other 30. The sum goes through the logistic function to 0.026 or 0.993.

That is not a bug in the training. It comes from the data. Here is rate_ewma per profile in the training rows:

Profileminmedianmax
benign1.0001.0001.000
scraper1.0541.2101.527
scanner1.2661.4851.824
flood23.10731.05140.243

Every benign row is exactly 1.0. The average only moves when a window has a request. So a client that never sends two requests in the same second stays at 1.0 forever. The scraper’s 1 s sleep has 50 ms of jitter. Now and then, two of its requests land in one window, and the average creeps up. XGBoost found the cheapest separation: “did this client ever send two requests within one second”.

Real visitors are messier. A person can click twice within a second, or a page can make two API calls for one action. Either one moves the average from 1.0 to 1.1. The client then scores 0.993 for its next six requests, until the 10% steps bring it back under 1.054. In enforce mode, a model trained on my synthetic benign profile throttles real people.

XGBoost 3 model format

XGBoost 3 saves base_score as a string that holds an array, "[7.8531075E-1]". Older versions wrote a plain number. Before I deployed, I made sure that 0.19.1 reads it. A local docker run of the image with the model in [ml] started, and six requests gave dvarapala_ml_score_count 6.

The model in the gateway

The model is a file in a ConfigMap, mounted through the operator’s gateway.mounts:

configMapGenerator:
  - name: dvarapala-model-v1
    files:
      - model.json=dvarapala-model.json
[ml]
model = "/etc/dvarapala-mounts/model/model.json"
threshold = 1.0

[[ml.llm.triggers]]
on = "ml_band"
band = [0.5, 1.0]

The gateway reads the model once at start, so the ConfigMap name carries a version. A new model means a new name, a changed pod spec and a rolled pod.

threshold = 1.0 is the watch-only setting. A score is never above 1, so the gateway throttles nothing, and dvarapala_ml_score records every score. The ml_band trigger sends clients with a score between 0.5 and 1.0 to the LLM. So the model’s suspects get a second opinion that I can read in the console. I removed the temporary route and its trigger in the same change.

What the Scores Look Like

A five minute rerun of the same load against the gateway with the model:

dvarapala_ml_score{quantile="0"} 0.026153777535046534
dvarapala_ml_score{quantile="0.5"} 0.9929251404657734
dvarapala_ml_score{quantile="0.99"} 0.9929251404657734
dvarapala_ml_score_count 19566

Two values, as trained. The ml_band trigger picked all 24 attacking clients and none of the benign ones:

Profileml_band samplesLLM labelMean confidence
scraper8scanner0.42
scanner8scanner0.82
flood8scanner0.52

The LLM is least sure about the scraper. That makes sense: it sends valid requests with a plausible user agent, and only the pace is wrong.

Turning it on

To see the other half work, I set threshold = 0.9 for one more five minute run of the same load. Then I put it back to 1.0:

ProfileWatching (1.0)Enforcing (0.9)
benign282 x 200267 x 200
scraper2285 x 200141 x 200, 2154 x 429
scanner2057 x 404, 492 x 40030 x 404, 271 x 400, 2256 x 429
flood14682 x 200, 18798 x 4298 x 200, 33592 x 429

The token bucket never touched the scraper, but now it lost 94% of its requests to 429. The 141 that got through, about 18 per client, are most likely the requests before each client first landed two in one second. The scanner’s probes stopped reaching the app. The flood lost the requests that the bucket let through between bursts before. Benign clients never saw a 429.

So enforcement does what the configuration reference says. I still set it back to 1.0, because of what the table in the training section shows. The benign profile here never double clicks, and real people do.

Lessons

  1. The model learns whatever separates your data most cheaply. With synthetic profiles that differ mostly in pace, that is one rate threshold. Make the benign traffic as varied as the attacks before you trust more than that.
  2. Look at the feature values behind a split. “rate_ewma < 1.054” reads like “about one request a second”. The rows show that it means “never two requests in one second”. A real visitor breaks that with one double click.
  3. LLM labels are good enough for “attack or not”. They had 93.2% agreement on the binary target, and they were poor on the type. Train on the binary label and keep the confidence filter.
  4. Moving averages remember. status_4xx_share carried a bad first run into labels minutes later. Restart the load, or wait, after a mistake.
  5. Watch before you enforce. threshold = 1.0 plus ml_band gives the score distribution and a second opinion on every suspect. No visitor is affected.

References