With multiple benchmarks confirming that at least one frontier model can autonomously execute an end-to-end compromise, cybersecurity experts are warning that companies have to pick up the pace in securing their attack surfaces and adopting AI-speed defenses.
On Sept. 2, consulting firm Booz Allen became the latest organization to confirm that a frontier model — Anthropic’s Mythos 5 — can act as a fully autonomous hacker and compromise a production-grade enterprise network. The company also released a metric, dubbed the Cyber Weapon Index (CWI), to benchmark a model’s capabilities, pairing the ability to find and exploit vulnerabilities with the ability to execute and attack a target.
In their evaluation, Mythos scored an 80, while the next-closest contender, SpaceXAI‘s Grok-4.5, scored a 49. However, the current standings are not as important as where we will be in six months, says Brad Medairy, president of Booz Allen’s National Cyber practice.
“Our view is there’s going to be some level of parity, at least between the frontier models and the Chinese models over probably a six-month horizon, and so everyone is in a rapid race to evolve these capabilities,” he says, adding that, when facing human attackers, defenders had the advantage, but “in the future, the balance of power shifts to the offense, because the attacker can deliver effects at speed and scale that the traditional defenses can’t keep pace with.”
Booz Allen’s findings mirror those of other efforts to benchmark the cyberattack capabilities of frontier models. In June, the AI Security Institute — a research arm of the UK government’s Department for Science, Innovation, and Technology — reported that capture-the-flag contests confirmed that Mythos could complete an end-to-end attack chain. In addition, OpenAI’s GPT-5.5 model completed the 32-step attack chain test. Both models only succeeded in less than half their attempts — 3 out of 10 for Mythos and 2 out of 10 for GPT-5.5.
Human-Speed Not Good Enough
A July attack on Taiwanese government servers by a Chinese-speaking cyber-threat group demonstrates the advantage of autonomous — or in that case, near-autonomous — hacking operations. The attack took place over a four-day window, compressing many of the reconnaissance and attack-execution steps.
“The agents chose which systems to map, which techniques to pull from public sources, and when to expand into new sectors, all without step-by-step human direction,” cybersecurity firm Tenable stated in an analysis of that incident and a half-dozen other similar AI-powered attacks.
The lesson for companies is that human-speed cybersecurity operations will not be enough in the future, says Medairy. A four-hour response time today may be considered reasonable — even exceptional — but that will not be a fast-enough reaction in the future.
“In the past, when you’re dealing with an adversary with a human behind the terminal, your defenses could outpace them,” he says. “An adversary that’s an agent operating at scale can outpace the defenses.”
While evaluating current frontier models shows the leading edge of attack automation, what really will accelerate attackers’ capabilities will not be frontier models, but open-weight models that make such autonomous attacks less costly, Nico Waisman, chief information security officer (CISO) at offensive cybersecurity vendor XBOW, tells Dark Reading.
“Open-weight models have gotten materially better at cyber, and that’s what moves the ROI calculation,” he says. “You no longer need frontier access to do this. That’s the point where automation becomes the cheaper option, not just the impressive one.”
Improved Attacks Rely on Stealth, Better Harnesses
Combining the token costs, relative success rates, and the CWI could give companies a good idea of how cost-effective autonomous attacks could be. However, another factor that is not accounted for is how stealthy such attacks are, Waisman says.
“Today’s models are extremely noisy,” he says. “They weren’t built to be quiet, and in offensive operations noise means early detection. That’s what makes an attacker think twice before turning an agent loose on a real target.”
A lot of the progress will boil down to development of more effective harnesses, not improvement in the models. XBOW, for example, used off-the-shelf models combined with human expertise and their own harnesses to find six vulnerabilities in Google’s Chrome, turning them into two attack chains, Waisman says.
“The frontier model was not our differentiator — we did not have a special one,” he says. “The harness and the people were the differentiator.”
Deception Could Be a Stopgap Measure
The long-term strategy for companies should be to understand their critical assets and protect them from unauthorized access. Companies should continue to patch, but vulnerabilities are no longer the unit of work because everyone has them, and no one can close them fast enough, Waisman says.
“Design to contain,” he says. “Use AI to automate every step of your defense that can be automated, from detection engineering through triage to incident response, because a defense running at human speed against an attack running at machine speed loses on arithmetic alone, no matter how good your people are.”
Autonomous attacks are the endpoint in an asymmetric equation that favors attackers. In the short term, companies should also look for asymmetric defenses, says Medairy. The company has a defensive approach, dubbed Guile, that presents false leads and dead ends — bait that AI systems are likely to swallow. While human attackers rarely fall for such deception, AI models fail more than 90% of the time, he says.
“It can beat an agent, but not a human,” Medairy says. “It just shows that we need to think about these more asymmetric approaches in the future, because we’ve built models, we’ve built technologies, we’ve built processes to defeat human attackers, and now we’re fighting machines and we’ve got to fundamentally change that game.”
Comments are closed