Over the last few months we have spent a lot of time on the hidden cost of retries during a flood. This is what we learned.
Latency budget
Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.
That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.
The numbers
Across the last quarter, 71% of challenged clients never attempted a solution, 24% solved one challenge and then behaved normally, and 5% solved challenges repeatedly while continuing to attack — the last group is where analysts spend their time.
Median added latency for legitimate visitors that were challenged was 280 ms on desktop and 410 ms on mid-range Android devices.
Testing in production, safely
Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.
This one feature has prevented more incidents than any detection we have ever shipped.
Compliance is a side effect
Regulators increasingly ask for evidence of resilience, not just promises. An incident timeline with start, peak, vectors and impact is exactly the evidence DORA and NIS2 ask for — and it falls out of good observability for free.
We export incident reports in formats auditors can file without anyone rewriting them.
What actually happens during a flood
The first thing to fail during a layer 7 flood is almost never bandwidth. It is connection slots, worker processes or database connections on the origin — resources measured in hundreds or thousands, not gigabits. An attacker who can make each request expensive only needs a few thousand requests per second.
That is why we score requests before they are proxied. By the time a request reaches your origin, it has been attributed to a client, compared against that client’s history and weighed against the current load on the route it targets.
Observability first
Every mitigation decision is logged with its reasons, and every log line can be traced to the rule and score components that produced it. When a customer asks why a request was challenged, the answer is a link, not a guess.
Logs stream to the customer’s SIEM within seconds, which also means their security team sees attacks in the same tools they use for everything else.
As always, questions and corrections are welcome at [email protected].