This post is about botnets built from misconfigured IoT cameras, again. It started, as most of our posts do, with an incident that did not go the way we expected.
What we changed
We moved the decision from a single threshold to a continuous score, added an explanation to every decision and made every rule testable against historical traffic before it goes live.
The result is fewer late-night pages for our analysts and — more importantly — fewer real users challenged by mistake.
Lessons for your own runbook
Know who can change DNS at two in the morning. Know your origin IPs and who can rotate them. Know which routes are expensive, and have a rate limit ready for each of them.
Most outages during attacks are not caused by the attack itself but by rushed changes made while under pressure.
What actually happens during a flood
The first thing to fail during a layer 7 flood is almost never bandwidth. It is connection slots, worker processes or database connections on the origin — resources measured in hundreds or thousands, not gigabits. An attacker who can make each request expensive only needs a few thousand requests per second.
That is why we score requests before they are proxied. By the time a request reaches your origin, it has been attributed to a client, compared against that client’s history and weighed against the current load on the route it targets.
Challenges beat blocks
Blocking lists go stale within minutes when attackers rotate through residential proxies. Proof-of-work does not care where a request comes from; it only cares whether the client is willing to pay the cost.
For a real browser that cost is a few hundred milliseconds, once per session. For a botnet sending a million requests a minute it is a million puzzles a minute — and at that point the attack stops being cheap.
Testing in production, safely
Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.
This one feature has prevented more incidents than any detection we have ever shipped.
Protecting the origin
None of this matters if the attacker can reach your origin directly. Historical DNS records, certificate transparency logs and misconfigured subdomains leak origin addresses all the time.
Allow-list the edge ranges, enable authenticated origin pulls and treat any origin IP that ever appeared in public DNS as burned.
The economics behind it
A booter service rents out a 100 Gbps attack for less than the price of a pizza. Defending against it with bandwidth alone is a race you lose. Defending against it by making each malicious request cost more than it earns is a race you win.
This is the entire idea behind never metering attack traffic: our costs scale with filtering, not with your invoice.
If you run into any of this in your own environment, our SOC is happy to take a look — even if you are not a customer.