Keeping search engine bots happy while blocking scrapers (part 4)

Here is a question we could not answer well a year ago: what really happens with keeping search engine bots happy while blocking scrapers? We can answer it now.

Measuring success

We track three numbers for every incident: time to mitigation, the share of attack traffic that reached the origin, and the share of legitimate traffic that was challenged. The first should be under a second, the second under 0.1% and the third under 1%.

Those numbers go into every incident report, and they are the same numbers we are measured against in our SLA.

Lessons for your own runbook

Know who can change DNS at two in the morning. Know your origin IPs and who can rotate them. Know which routes are expensive, and have a rate limit ready for each of them.

Most outages during attacks are not caused by the attack itself but by rushed changes made while under pressure.

Why per-route baselines matter

A thousand requests per second to your homepage is Tuesday. A thousand requests per second to your password-reset endpoint is an attack. Global rate limits cannot tell the difference; per-route baselines can.

We learn the normal shape of traffic per route and per hour of the week, so a surge on a sensitive endpoint raises the risk score long before it approaches a global threshold.

Challenges beat blocks

Blocking lists go stale within minutes when attackers rotate through residential proxies. Proof-of-work does not care where a request comes from; it only cares whether the client is willing to pay the cost.

For a real browser that cost is a few hundred milliseconds, once per session. For a botnet sending a million requests a minute it is a million puzzles a minute — and at that point the attack stops being cheap.

Latency budget

Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.

That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.

What we changed

We moved the decision from a single threshold to a continuous score, added an explanation to every decision and made every rule testable against historical traffic before it goes live.

The result is fewer late-night pages for our analysts and — more importantly — fewer real users challenged by mistake.

Testing in production, safely

Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.

This one feature has prevented more incidents than any detection we have ever shipped.

The full incident data behind this post is available to customers in the dashboard under Reports.