Rate limiting APIs without breaking legitimate integrations (part 2)

Here is a question we could not answer well a year ago: what really happens with rate limiting APIs without breaking legitimate integrations? We can answer it now.

Protecting the origin

None of this matters if the attacker can reach your origin directly. Historical DNS records, certificate transparency logs and misconfigured subdomains leak origin addresses all the time.

Allow-list the edge ranges, enable authenticated origin pulls and treat any origin IP that ever appeared in public DNS as burned.

Measuring success

We track three numbers for every incident: time to mitigation, the share of attack traffic that reached the origin, and the share of legitimate traffic that was challenged. The first should be under a second, the second under 0.1% and the third under 1%.

Those numbers go into every incident report, and they are the same numbers we are measured against in our SLA.

Latency budget

Our budget for the whole filtering pipeline is one millisecond at the 99th percentile. Anything that cannot be decided within that budget runs asynchronously and influences the next request from the same client, not the current one.

That constraint shapes everything: data structures, where state lives and which signals we are willing to compute inline.

Observability first

Every mitigation decision is logged with its reasons, and every log line can be traced to the rule and score components that produced it. When a customer asks why a request was challenged, the answer is a link, not a guess.

Logs stream to the customer’s SIEM within seconds, which also means their security team sees attacks in the same tools they use for everything else.

What actually happens during a flood

The first thing to fail during a layer 7 flood is almost never bandwidth. It is connection slots, worker processes or database connections on the origin — resources measured in hundreds or thousands, not gigabits. An attacker who can make each request expensive only needs a few thousand requests per second.

That is why we score requests before they are proxied. By the time a request reaches your origin, it has been attributed to a client, compared against that client’s history and weighed against the current load on the route it targets.

Lessons for your own runbook

Know who can change DNS at two in the morning. Know your origin IPs and who can rotate them. Know which routes are expensive, and have a rate limit ready for each of them.

Most outages during attacks are not caused by the attack itself but by rushed changes made while under pressure.

The full incident data behind this post is available to customers in the dashboard under Reports.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top