The debate around AI governance heated up this past week as AI experts warned that the growing number of rogue AI agents and instances of misaligned behavior are merely a preface to worse threats in the future.

On Sept. 12, Anthropic CEO Dario Amodei warned in a widely read column that AI could soon slip the harness of human control if development is left unchecked. Within a few days, Microsoft delivered a Humanist AI Code of Conduct (read: manifesto) pledging human-centered development, and Google DeepMind’s CEO agreed in principle in a post on X to the calls for more controls. Although Anthropic, OpenAI, and x.AI have all also suggested tighter government regulations for the technology, President Donald Trump dismissed the idea, suggesting that winning the AI competition with China is more important, reportedly saying, “Whoever wins AI, wins.”

In his column, Anthropic’s Amodei argued that model makers and AI development labs have a narrow window to refocus on safety and security.

Related:Need for Speed: AI-Driven Attacks Are Changing Security Strategies

“The current models are an almost endless goldmine of insight into both how to build AI well, and what can sometimes go wrong with it if it isn’t built well,” he wrote. “I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong.”

In the meantime, companies are left to their own devices when it comes to implementing safety protocols for their agentic and autonomous AI programs. The incidents and the calls for regulations highlight the gaps between quickly advancing capabilities and the industry’s ability to prevent harm, Ramy Rahman, senior principal solutions engineer at ArmorCode, tells Dark Reading.

“‘Move fast and ask forgiveness later’ is not an acceptable approach when organizations bear the consequences,” he says. “Those developing and deploying AI should be accountable for the risks they introduce.”

Time for a Pause in AI Capabilities?

The concerns are rolling out as more reports of AI agents circumventing security and safety controls come to light. Meta, for example, recently acknowledged that its own AI model, Muse Spark 1.1, essentially followed the same playbook as OpenAI’s model, escaping its sandbox during security testing to compromise another company’s server.

Anthropic has faced similar incidents: In three cases (albeit out of more than 141,000), Claude gained access to the Internet in tests during which the model was thought to be contained. And on Sept. 16, OpenAI revealed another six types of misalignment that the company detected during model tests, including generating new instructions to avoid constraints and searching for exposed API keys to fabricate data.

Related:Trump, Tech Giants Strike Voluntary AI Safety Accord

National governments also see a variety of threats in the development of AI. China spy chief Chen Yixin — officially the head of China’s Ministry of State Security — warned in a September article that AI could be a threat to Communist Party rule, and that the technology could be used for effective espionage campaigns against China.

South Korea, which has already passed laws to curb the use of deepfakes during political campaigns, plans to revise its AI security guidelines to focus on issues that could arise as companies deploy agentic AI services. The country already passed its AI Basic Law in 2025, which went into effect on Jan. 22 this year; it requires companies to offer guidance and protective measures for their AI offerings.

It’s unclear whether AI model makers truly fear the machinations of the AI agents they are creating or just fear being held liable for rogue behavior. Google DeepMind, Microsoft, and xAI, however, all agreed in principle with a potential pause, the institution of some regulations, and the creation of an independent testing regime.

Related:Nvidia Launches AI Agent Safety Platform to Prevent Rogue Activities

Using the US Financial Industry Regulatory Authority (FINRA) as a model, a new regulatory agency could assess development protocols and conduct independent testing to minimize risks, argued Demis Hassabis, co-founder and CEO of Google DeepMind, in a July column.

“The rapid progress we’re seeing in AI requires a new approach to testing frontier AI model capabilities that is dynamic, adaptable, and rigorous,” he wrote. “The US is well-positioned, given its economic and technical standing, to take the first step in developing such a framework.”

In its Humanist AI Code of Conduct released on Sept. 14, Microsoft pledged to maintain human control of AI systems and to stay within scope.

“[W]e welcome the research, focus, and deliberate pacing needed to get alignment right as the design goal,” Microsoft CEO Satya Nadella stated in a post to X on Sept. 13. “We also welcome ideas like ’embedded evaluators’ and the broader efforts to develop the mechanisms to make this more than just talk.”

Yet not all companies are seeking government oversight. At the other end of the debate is Meta. The government does not need to intervene because market forces will demand properly aligned AI agents, Mark Zuckerberg, Meta’s CEO, stated in his own post on X. But using independent evaluators to test security and safety should become an industry-standard practice, he said, claiming that the company delayed its own AI, Muse, to focus on security and safety.

“My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models,” he stated. “Any lab that doesn’t focus on alignment will fall behind.”

As mentioned, the Trump administration has also pushed back on any limits to AI development, citing the need to compete with China and other countries.

Assume AI Liability Is Yours

For now, the immediate risks are not world-ending catastrophes but business disruption and liability concerns. In its “AI Risk and Resilience” report, Google outlined a number of threats, from new AI techniques used by cyberattackers to failures caused by enterprises’ lack of AI governance. In one “denial-of-wallet” incident, for example, a company failed to limit an agent, resulting in the agent costing the company $50,000 and halting business transactions when a logic failure caused the software to repeatedly call an expensive API.

“Many organizations now have AI policies, acceptable use guidelines, and a defined risk appetite — what they often lack is a way to enforce and verify those policies in practice,” says Muhammad Muneer, technical manager at Google Cloud’s Mandiant Consulting group, pointing to limited logging of AI activity, missing controls around what agents can access, and little visibility into how and where AI is used as common issues.

“A policy is only effective if you can see whether it’s being followed,” he says.

Companies should not wait for frontier model providers to secure their agentic AI offerings but focus on making sure that they have good visibility into what actions their AI agents are taking and whether malicious or rogue agents are acting in their environment, experts say.

Using AI models to test your own defenses and find vulnerabilities in your software and networks can help companies reduce their attack surface area because otherwise, attackers will do it for free, says Jack Nelson, CISO of Ivanti, a provider of autonomous endpoint management.

“At an enterprise level, it gets back to pretty foundational cybersecurity issues [such as] identity, auditing, logging, to be able to see not just … who’s using the model,” he says, “but also what is the model itself doing and then what are the parameters of the model’s activity.”

In the end, he adds, the worry for enterprises is “less this idea of robots taking over the Internet and more the idea of misuse or overuse internally, the over-creation of slop, or just grinding away on tokens on a problem that someone never thought to tell the model, ‘Will you ask me if you run into a roadblock?'”





Source link

#

Comments are closed