Designing a risk score that analysts can explain (part 2)

This post is about designing a risk score that analysts can explain. It started, as most of our posts do, with an incident that did not go the way we expected.

The numbers

Across the last quarter, 71% of challenged clients never attempted a solution, 24% solved one challenge and then behaved normally, and 5% solved challenges repeatedly while continuing to attack — the last group is where analysts spend their time.

Median added latency for legitimate visitors that were challenged was 280 ms on desktop and 410 ms on mid-range Android devices.

Testing in production, safely

Every rule starts in log mode. We replay the previous seven days of traffic through it and show exactly which requests it would have affected before anyone can promote it to challenge or block.

This one feature has prevented more incidents than any detection we have ever shipped.

Protecting the origin

None of this matters if the attacker can reach your origin directly. Historical DNS records, certificate transparency logs and misconfigured subdomains leak origin addresses all the time.

Allow-list the edge ranges, enable authenticated origin pulls and treat any origin IP that ever appeared in public DNS as burned.

Measuring success

We track three numbers for every incident: time to mitigation, the share of attack traffic that reached the origin, and the share of legitimate traffic that was challenged. The first should be under a second, the second under 0.1% and the third under 1%.

Those numbers go into every incident report, and they are the same numbers we are measured against in our SLA.

Observability first

Every mitigation decision is logged with its reasons, and every log line can be traced to the rule and score components that produced it. When a customer asks why a request was challenged, the answer is a link, not a guess.

Logs stream to the customer’s SIEM within seconds, which also means their security team sees attacks in the same tools they use for everything else.

The full incident data behind this post is available to customers in the dashboard under Reports.