Anthropic CEO Dario Amodei has warned that increasingly powerful AI systems cannot be safely entrusted to governments or corporations alone, arguing that both require meaningful checks and balances as artificial intelligence becomes more capable.
In an essay published Wednesday, Amodei also said governments should have the authority to block or reverse deployments of advanced AI models that pose unacceptable risks, including threats related to cybersecurity, biological weapons and loss of control of AI systems.
The Anthropic CEO argued that AI companies could eventually develop “quasi-state characteristics,” becoming powerful enough to influence society in ways traditionally associated with governments. He wrote that advanced AI may become “so powerful that it cannot safely be fully entrusted to either governments or companies.”
Why Transparency Is No Longer Enough
Amodei said existing transparency-focused approaches to AI oversight are no longer sufficient as frontier models become increasingly capable.
He called for “more serious and binding regulation of AI” and compared advanced AI systems to airplanes and pharmaceuticals, arguing that they should undergo mandatory third-party testing before deployment.
“Their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety,” Amodei wrote.
The proposed evaluations would focus on risks related to cybersecurity, biological weapons, loss of control and automated research capabilities. He added that AI developers should maintain strong security standards, conduct regular testing and promptly report major safety incidents.
Preparing For AI-Driven Job Loss
Beyond safety risks, Amodei warned that AI could create larger and more persistent labor-market disruptions than previous technological shifts.
“It's reasonable to think that AI could produce much larger disruptions to the labor market than previous technologies,” he wrote.
To address those risks, Amodei proposed measures including wage insurance, worker-retention incentives, workforce training programs and expanded monitoring of AI’s impact on employment. He also suggested broader income-support mechanisms could eventually become necessary if displacement becomes widespread.
Amodei argued that policymakers should focus on ensuring the benefits of AI-driven economic growth are shared broadly rather than concentrated among a small group of companies and investors.
Who Should Control Powerful AI?
The essay also addressed concerns about AI’s impact on civil liberties, political power and national security.
Amodei warned that advanced AI could become a tool of surveillance and authoritarian control if governments gain unchecked access to increasingly capable systems. He also called for stronger protections against domestic use of fully autonomous weapons.
At the same time, he argued that private companies should not hold unchecked influence over increasingly powerful AI systems, warning that some firms could eventually develop “quasi-state characteristics.”
On geopolitics, Amodei described AI as a technology comparable in strategic significance to nuclear weapons, arguing that it could become a dominant source of military and economic power. He called for closer coordination among democratic nations on AI policy, safety standards and supply-chain controls.
Mythos Sparks Concern
The debate around Mythos follows months of friction between Anthropic and U.S. officials over AI safety and military applications. Earlier this year, the Trump administration moved to phase out Anthropic’s technology from federal agencies after the company maintained restrictions on mass surveillance and autonomous weapons.
Anthropic later briefed members of the House Homeland Security Committee on Mythos and its national security implications. Pentagon tech chief Emil Michael also described the model as a “separate national security moment” because of its advanced cyber capabilities.
The essay comes days after Anthropic launched Mythos 5, the latest version of its cybersecurity-focused AI model, alongside Claude Fable 5, its newest consumer-facing system.
Against that backdrop, Amodei’s call for stronger safeguards, mandatory testing and government authority to block dangerous AI deployments arrives as policymakers and technology companies grapple with increasingly powerful frontier AI systems.
Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors.
Image via Shutterstock
Login to comment