Keeping search engine bots happy while blocking scrapers (part 3)

Here is a question we could not answer well a year ago: what really happens with keeping search engine bots happy while blocking scrapers? We can answer it now.

What actually happens during a flood

The first thing to fail during a layer 7 flood is almost never bandwidth. It is connection slots, worker processes or database connections on the origin — resources measured in hundreds or thousands, not gigabits. An attacker who can make each request expensive only needs a few thousand requests per second.

That is why we score requests before they are proxied. By the time a request reaches your origin, it has been attributed to a client, compared against that client’s history and weighed against the current load on the route it targets.

Protecting the origin

None of this matters if the attacker can reach your origin directly. Historical DNS records, certificate transparency logs and misconfigured subdomains leak origin addresses all the time.

Allow-list the edge ranges, enable authenticated origin pulls and treat any origin IP that ever appeared in public DNS as burned.

Challenges beat blocks

Blocking lists go stale within minutes when attackers rotate through residential proxies. Proof-of-work does not care where a request comes from; it only cares whether the client is willing to pay the cost.

For a real browser that cost is a few hundred milliseconds, once per session. For a botnet sending a million requests a minute it is a million puzzles a minute — and at that point the attack stops being cheap.

Testing in production, safely

Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.

This one feature has prevented more incidents than any detection we have ever shipped.

As always, questions and corrections are welcome at [email protected].