khalaf.ai

Mythos will leave us all cold and exposed, or so we're told.

The most tempered conclusion we can draw from Anthropic's announcement is that we're going to be bombarded with more "critical vulnerabilities". But let's step back and think about what that means.

A real crisis

I started my career building security products for AWS S3, earth's largest system. The big Log4j (Log4Shell) vulnerability was unearthed in my first weeks there. Company-wide panic ensued as every team had its oncall engineers on all-nighters to assess & remediate.

The panic was justified: Log4j was ubiquitous, easy to exploit with string injection, and the exploit would allow attackers full Remote Code Execution (RCE). The ultimate nightmare scenario.

Yet after that incident, thousands of vulnerabilities have crossed my eyes often with little consequence.

The (not so) new bottleneck

If AI agents can scan codebases better than ever, then that means more vulnerabilities than ever. You'd think this breaks the pipeline of vulns, but it was broken to begin with.

There's a whole industry born out of vulnerability detection & prioritization. You've probably heard some of these terms: Tenable, OWASP, Qualys, CVSS, to name a few.

How do we score what's important? CVSS assigns a severity score based on attack complexity, privileges required, and system exposure. Anything 7 or above is "High or Critical". This results in over half of vulnerabilities crossing the High threshold.

In practice, this leads to a ton of random noise.

Log4Shell was bad because it was:

• Widespread
• Easy to exploit
• Catastrophic (RCE)

How do we filter all this noise while also being aware of the actual consequential ones? Anthropic points to a better model.

EPSS

Exploit Prediction Scoring System (EPSS) takes a different approach entirely. Rather than describing a vulnerability's characteristics, it predicts the probability that a given CVE will be exploited in the next 30 days. Trained on real-world exploitation data, and updated daily.

Under the hood, EPSS v4 is built on XGBoost. No next-token prediction, no fitting transformers into the solution.

The results speak for themselves. At a 5.5% threshold, EPSS achieves the same 74.7% coverage of exploited CVEs as CVSS 7+, while flagging only 5.9% of all CVEs instead of 50.7%. Efficiency jumps from 5.9% to 47.3%.

The same protection with ~90% less noise.

Credit: Empirical Security

Local Models to the Rescue

EPSS is a universal model. It doesn't know your codebase, your team's remediation velocity, or what a "critical" component actually means in your stack. Severity is contextually defined.

The teams that will win are the ones who tune prioritization models to their own environment: vuln age, domain, blast radius, historical remediation patterns. Local models trained on org-specific data can encode all of that.

The pipeline starts to look like this: coding agents surface vulnerabilities at scale, local probabilistic models rank them by what actually matters to your org, and engineers get scoped remediation instructions rather than a queue of 10,000 CVEs.

LLMs belong at the edges of this pipeline. Not as the judgment layer.

Thanks

This piece was inspired by a conference talk I accidentally sat in on from Michael Roytman (@mroytman). Worth following his work if this space interests you.