Building a per-route baseline without storing every request

Over the last few months we have spent a lot of time on building a per-route baseline without storing every request. This is what we learned.

Observability first

Every mitigation decision is logged with its reasons, and every log line can be traced to the rule and score components that produced it. When a customer asks why a request was challenged, the answer is a link, not a guess.

Logs stream to the customer’s SIEM within seconds, which also means their security team sees attacks in the same tools they use for everything else.

What we got wrong

Our first version challenged too eagerly on mobile networks, where thousands of real users share a handful of carrier-grade NAT addresses. Reputation that is shared is reputation that is noisy.

We now weight fingerprint consistency and session behaviour far more heavily than IP reputation for traffic from known mobile carrier ranges.

The numbers

Across the last quarter, 71% of challenged clients never attempted a solution, 24% solved one challenge and then behaved normally, and 5% solved challenges repeatedly while continuing to attack — the last group is where analysts spend their time.

Median added latency for legitimate visitors that were challenged was 280 ms on desktop and 410 ms on mid-range Android devices.

Latency budget

Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.

That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.

Why per-route baselines matter

A thousand requests per second to your homepage is Tuesday. A thousand requests per second to your password-reset endpoint is an attack. Global rate limits cannot tell the difference; per-route baselines can.

We learn the normal shape of traffic per route and per hour of the week, so a surge on a sensitive endpoint raises the risk score long before it approaches a global threshold.

We will follow up with the numbers from the next quarter.

4 thoughts on “Building a per-route baseline without storing every request”

  1. We moved from a scrubbing provider to always-on last year; time to mitigation went from minutes to basically nothing.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top