Scraper bots are getting better at pretending to be Chrome

Over the last few months we have spent a lot of time on scraper bots are getting better at pretending to be Chrome. This is what we learned.

Latency budget

Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.

That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.

Compliance is a side effect

Regulators increasingly ask for evidence of resilience, not just promises. An incident timeline with start, peak, vectors and impact is exactly the evidence DORA and NIS2 ask for — and it falls out of good observability for free.

We export incident reports in formats auditors can file without anyone rewriting them.

What actually happens during a flood

The first thing to fail during a layer 7 flood is almost never bandwidth. It is connection slots, worker processes or database connections on the origin — resources measured in hundreds or thousands, not gigabits. An attacker who can make each request expensive only needs a few thousand requests per second.

That is why we score requests before they are proxied. By the time a request reaches your origin, it has been attributed to a client, compared against that client’s history and weighed against the current load on the route it targets.

The economics behind it

A booter service rents out a 100 Gbps attack for less than the price of a pizza. Defending against it with bandwidth alone is a race you lose. Defending against it by making each malicious request cost more than it earns is a race you win.

This is the entire idea behind never metering attack traffic: our costs scale with filtering, not with your invoice.

{ "match": { "path": "/api/v1/checkout", "risk": ">= 46" }, "action": "challenge" }

If you run into any of this in your own environment, our SOC is happy to take a look — even if you are not a customer.