Introducing adaptive proof-of-work difficulty

This post is about introducing adaptive proof-of-work difficulty. It started, as most of our posts do, with an incident that did not go the way we expected.

The numbers

Across the last quarter, 71% of challenged clients never attempted a solution, 24% solved one challenge and then behaved normally, and 5% solved challenges repeatedly while continuing to attack — the last group is where analysts spend their time.

Median added latency for legitimate visitors that were challenged was 280 ms on desktop and 410 ms on mid-range Android devices.

Challenges beat blocks

Blocking lists go stale within minutes when attackers rotate through residential proxies. Proof-of-work does not care where a request comes from; it only cares whether the client is willing to pay the cost.

For a real browser that cost is a few hundred milliseconds, once per session. For a botnet sending a million requests a minute it is a million puzzles a minute — and at that point the attack stops being cheap.

Lessons for your own runbook

Know who can change DNS at two in the morning. Know your origin IPs and who can rotate them. Know which routes are expensive, and have a rate limit ready for each of them.

Most outages during attacks are not caused by the attack itself but by rushed changes made while under pressure.

Testing in production, safely

Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.

This one feature has prevented more incidents than any detection we have ever shipped.

What we got wrong

Our first version challenged too eagerly on mobile networks, where thousands of real users share a handful of carrier-grade NAT addresses. Reputation that is shared is reputation that is noisy.

We now weight fingerprint consistency and session behaviour far more heavily than IP reputation for traffic from known mobile carrier ranges.

{ "match": { "path": "/api/v1/checkout", "risk": ">= 60" }, "action": "challenge" }

We will follow up with the numbers from the next quarter.