The hidden cost of retries during a flood (part 3)

Over the last few months we have spent a lot of time on the hidden cost of retries during a flood. This is what we learned.

Measuring success

We track three numbers for every incident: time to mitigation, the share of attack traffic that reached the origin, and the share of legitimate traffic that was challenged. The first should be under a second, the second under 0.1% and the third under 1%.

Those numbers go into every incident report, and they are the same numbers we are measured against in our SLA.

Why per-route baselines matter

A thousand requests per second to your homepage is Tuesday. A thousand requests per second to your password-reset endpoint is an attack. Global rate limits cannot tell the difference; per-route baselines can.

We learn the normal shape of traffic per route and per hour of the week, so a surge on a sensitive endpoint raises the risk score long before it approaches a global threshold.

What we got wrong

Our first version challenged too eagerly on mobile networks, where thousands of real users share a handful of carrier-grade NAT addresses. Reputation that is shared is reputation that is noisy.

We now weight fingerprint consistency and session behaviour far more heavily than IP reputation for traffic from known mobile carrier ranges.

Protecting the origin

None of this matters if the attacker can reach your origin directly. Historical DNS records, certificate transparency logs and misconfigured subdomains leak origin addresses all the time.

Allow-list the edge ranges, enable authenticated origin pulls and treat any origin IP that ever appeared in public DNS as burned.

The economics behind it

A booter service rents out a 100 Gbps attack for less than the price of a pizza. Defending against it with bandwidth alone is a race you lose. Defending against it by making each malicious request cost more than it earns is a race you win.

This is the entire idea behind never metering attack traffic: our costs scale with filtering, not with your invoice.

Observability first

Every mitigation decision is logged with its reasons, and every log line can be traced to the rule and score components that produced it. When a customer asks why a request was challenged, the answer is a link, not a guess.

Logs stream to the customer’s SIEM within seconds, which also means their security team sees attacks in the same tools they use for everything else.

{ "match": { "path": "/search", "risk": ">= 62" }, "action": "challenge" }

If you run into any of this in your own environment, our SOC is happy to take a look — even if you are not a customer.