This post is about botnets built from misconfigured IoT cameras, again. It started, as most of our posts do, with an incident that did not go the way we expected.
Measuring success
We track three numbers for every incident: time to mitigation, the share of attack traffic that reached the origin, and the share of legitimate traffic that was challenged. The first should be under a second, the second under 0.1% and the third under 1%.
Those numbers go into every incident report, and they are the same numbers we are measured against in our SLA.
Challenges beat blocks
Blocking lists go stale within minutes when attackers rotate through residential proxies. Proof-of-work does not care where a request comes from; it only cares whether the client is willing to pay the cost.
For a real browser that cost is a few hundred milliseconds, once per session. For a botnet sending a million requests a minute it is a million puzzles a minute — and at that point the attack stops being cheap.
The numbers
Across the last quarter, 71% of challenged clients never attempted a solution, 24% solved one challenge and then behaved normally, and 5% solved challenges repeatedly while continuing to attack — the last group is where analysts spend their time.
Median added latency for legitimate visitors that were challenged was 280 ms on desktop and 410 ms on mid-range Android devices.
What we got wrong
Our first version challenged too eagerly on mobile networks, where thousands of real users share a handful of carrier-grade NAT addresses. Reputation that is shared is reputation that is noisy.
We now weight fingerprint consistency and session behaviour far more heavily than IP reputation for traffic from known mobile carrier ranges.
Latency budget
Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.
That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.
{ "match": { "path": "/wp-login.php", "risk": ">= 65" }, "action": "challenge" }
We will follow up with the numbers from the next quarter.
We had the exact false-positive issue with CGNAT carriers. The weighting change makes sense.
How does the proof-of-work challenge behave on older Android devices? Any numbers below Android 10?
Carpet bombing is nasty. Good to see a clear explanation of it.
Verified good bots and allow-listed partners bypass challenges entirely.
Our auditors asked for exactly this kind of incident evidence under DORA.
Could you share the dataset behind the percentages?
Is the risk score exposed in the logs so we can build our own dashboards on it?
The billing model is what got our finance team on board, honestly.
Thanks! Yes — the risk score and its components are included in every log record.
Carpet bombing is nasty. Good to see a clear explanation of it.
Is the risk score exposed in the logs so we can build our own dashboards on it?