In the mad dash to deploy agentic artificial intelligence (AI) technology, developers aren’t taking enough time to understand how their programs work, and they’re inadvertently generating a whole lot of very old-fashioned vulnerabilities.
The universe of AI agents in the advanced economies of today’s world is immeasurably large; literally, nobody has any clue how many of these things are out there. Some recent data suggests that somewhere around a third of organizations have either already adopted or will adopt, agentic AI tech soon, but even those measurements rest on self-reporting and generalized data, or loose predictions.
Contrary to popular belief, however, the agents themselves are not black boxes. In an unusually long presentation at Infosecurity Europe next month, researchers at Acronis are going to attempt to correct this unhelpful narrative by demonstrating how these bots work at a fundamental level. And by picking apart how AI agents work, they argue, an even more interesting finding emerges: that the cybersecurity vulnerabilities in this tech are not the fault of the AI; they’re mostly a byproduct of traditionally bad coding.
“What people don’t understand is that agentic systems still rely on a lot of old world technology and a lot of old world vulnerabilities,” says Acronis senior security researcher Eliad Kimhy. As agentic AI tech spreads more and more, “What we are going to see being abused are plain old vulnerabilities in software. And if you don’t understand that, you’re going to write bad software, and you’re going to rely on your large language model (LLM) to do the rest. That’s a bad approach.”
The Vulnerabilities in Agentic AI
Last fall, researchers discovered a critical vulnerability in Salesforce. If an attacker planted a malicious prompt in a certain kind of Salesforce form, an AI agent interpreting it on the back end might carry out its instructions. The issue was made worse by the fact that Salesforce was still whitelisting an expired, easily purchasable domain.
Early this year, a researcher discovered a dangerous exploit chain in ServiceNow. Thanks to an overly permissive chatbot — protected only by a factory default credential — that could be authenticated as any user simply by supplying their email address, the researcher found that he could access and create powerful AI agents in any company’s ServiceNow instance.
What do these stories, and so many more like it, have in common?
Since agentic AI has introduced so much new risk to organizations, one might reasonably assume that agentic AI technology is itself risky. But considering the sorts of vulnerabilities — lack of input sanitization, hardcoded credentials, insufficient access controls — what’s new and “intelligent” about any of that?
“I think the flashy thing — the fun types of hack, the types of hack that everybody wants to talk about — is jailbreaks. That’s not really the point of failure we need to think about,” Kimhy argues.
The more significant point of failure is more unique to agents themselves, Acronis says. It lives right at the intersection between the AI and the traditional software it interacts with. To understand why that intersection is so dangerous, one first needs a fundamental understanding of how AI agents work.
How AI Agents Work
“The problem is that, a lot of the time, people look at these agentic systems as a black box. They think, OK, there’s input, there’s some magic happening in the middle, and then there’s output — we don’t know what’s going on [in the middle]. The message that we’re interested in helping people understand is that it is not a black box,” Kimhy says.
From a zoomed out perspective, an AI agent can be thought of as a system of two halves. “It’s an ecosystem that includes, on one hand, deterministic systems which are tools, basically old world software. A function that takes an argument just like any other function, and produces a deterministic result. The tool that is connected to a non-deterministic system, which is the LLM. That LLM works by understanding probabilities. These two things together form a system,” Kimhy explains.
Crucially, it is in the juxtaposition of the deterministic and non-deterministic halves that most agentic vulnerabilities arise.
In their presentation, Kimhy and his colleague, Acronis lead security researcher Syed Aizad, will demonstrate how this works using a sample AI agent underpinning a travel booking platform. Using cutting-edge reasoning agents, connected to totally inoffensive tools, any number of vulnerabilities still arise. A user might ask for their booking information, for instance, and the agent might supply it to them without realizing that the user might be lying about who they are.
This is not the fault of the agent; it’s a simple matter of authentication. Researchers demonstrated this exact scenario last December, using a program powered by a Microsoft Copilot Studio agent to leak personally identifying information (PII).
How to Secure Agentic Technology
It would be perfectly straightforward to design an authentication check for an AI agent, of course. But are slapdash “Agentforce” or “Now Assist” agents, or increasingly common vibe coded programs, accounting for that and a thousand potentially other vulnerable interactions between those deterministic and non-deterministic halves?
“People are going to [deploy agentic AI] without a deep understanding of how these systems work, and how they’re connected to each other. And that’s incredibly important to understand. The fixes for the LLM itself are not the same as the fixes for the software,” he says, adding that “more specifically, a lot of focus is now on the non-deterministic side. That’s the sexy part. But that’s really only half the picture, maybe even less than half the picture.”
Kimhy’s conference catchphrase is “old world principles with a new world spin”: applying time-tested cybersecurity principles to this new tech. This includes preventing agents from leaking data with standard token-based authentication, or applying access controls to the AI, just as one would a human employee.
“We need to first incorporate old world thinking, to understand that [traditional software principles] have always been a part of this system, and these tools need to be considered,” he explains. “But we need to put a new world twist on it, because now we’ve connected [that software] to something that is unpredictable in a lot of ways. And that is something that I think there’s just not enough awareness of.”
Comments are closed