AI Developers: It Could Kill Us All Within 5 Years!

Anthropic CEO Dario Amodei has called on artificial intelligence companies to slow the pace of developing model capabilities amid growing concerns about the misuse of AI, proposing a three-step framework to regulate the speed of development and allow more time to address the risks associated with these technologies.

In a lengthy article published on the platform X on Saturday, Amodei wrote, “We must slow the pace of improving AI model capabilities. Progress will still appear rapid, and we must wisely use the time we gain.”

Elon Musk, who runs xAI, and Sam Altman, CEO of OpenAI, expressed agreement with Amodei in posts on X, according to Reuters.

Amodei’s three-step plan includes appointing independent reviewers within leading AI companies, coordinating among firms developing advanced AI models to establish safety standards and curb unregulated AI development, and fostering international cooperation to manage its risks.

Amodei published his article after Anthropic, headquartered in San Francisco, released a threat intelligence report on Thursday detailing how various entities used the company’s Claude AI models for activities ranging from weapons development and cyberattacks to surveillance and fraud.

Amodei pointed to AI’s growing ability to self-improve, highlighting long-standing concerns that this capability could surpass human control over its operation, as well as recent incidents involving OpenAI and Hugging Face, as primary reasons for calling for a brake on the advancement of model capabilities.

Concerns over the potential harms of AI intensified last week when Jacob Cookson, a researcher at Anthropic, resigned, stating that “people working on AI development firmly believe it could kill us all by the end of this decade.”

Several OpenAI officials suggested that leading laboratories should be prepared to coordinate on voluntary slowdowns if necessary to build trust in the safety measures they are implementing.

Anthropic has positioned itself as a more safety-conscious laboratory in the field of advanced AI models, but it is not immune to these concerns.

Last week, the company revealed another incident in which an AI model breached external systems, following a July incident where some of its Claude models breached the systems of three companies during cybersecurity tests.

Amodei wrote, “Given the accelerating pace of AI capability development, I fear that within six to twelve months, such a swarm could take control of the entire internet, potentially causing hundreds of billions of dollars in damage.”

Amodei clarified that he is not calling for a halt to model training or technological progress, but rather ensuring that companies allocate sufficient time to align and secure their models, and that third-party evaluators verify these steps.

However, significant financial stakes are tied to maintaining progress over competitors. Both OpenAI and Anthropic are preparing for major initial public offerings. Each new capability can help support future funding rounds, infrastructure commitments, or IPOs.

Regarding the proposed three-step framework, Amodei said that Anthropic would appoint permanent third-party reviewers within leading artificial intelligence companies, granting them access to relevant tools and internal risk assessment processes.

Reuters reported last week that a swarm of rogue OpenAI agents hijacked a German website, turning it into an advertisement board for other AI agents, while company officials kept details of ChatGPT’s development under wraps at a time when executives were dealing with the fallout from the July hack of the open-source software repository Hugging Face.

Amodei stated that AI companies should voluntarily collaborate to establish standards, as an increasing number of U.S. lawmakers call for new regulations to govern AI systems.

He added that a coordinated approach would enable leading U.S. AI companies to conduct necessary safety research and implement safeguards without placing them at a competitive disadvantage. He noted that this approach would likely require targeted exemptions from U.S. antitrust laws to allow cooperation in certain areas.

Amodei said that any slowdown by countries governed by democratic systems should be limited to maintaining the progress U.S. AI companies have over China, considering China’s AI superiority as a threat to U.S. national security.

Amodei wrote: “The pace of development within democracies will be determined by the progress U.S. companies achieve over authoritarian systems, foremost among them the Chinese Communist Party.”

For his part, Hani Nofal, Regional Technology President for the Middle East and Africa at NTT DATA, said that warnings about human extinction or AI going out of control lack any proven scientific basis.

In an interview with “Al Arabiya Business,” he added that technology inherently carries some risks. However, the real danger in the rapid evolution of technology lies in its malicious use, which requires strict controls.

He clarified that competition between companies and international hostility pose a significant challenge in determining who is responsible for regulating and controlling AI and testing models before deployment, and who will bear responsibility for potential breaches.

He pointed out that it is essential to differentiate how AI is handled based on the level of benefit derived from it; a model for summarizing text or sending emails cannot be compared to one used in banking transactions or in laboratories for developing biological viruses.

Dr. Mohamed Adel El-Beltagy, CEO of Optomateca, said that despite their differences, the agreement among prominent figures in the technology and AI sectors to warn about the rapid development of AI models reflects genuine concerns about the level of autonomy these models have begun to possess.

El-Beltagy, in an interview with “Al Arabiya Business,” referred to an incident that occurred last July, when an independent AI system was able to execute actions without clear direction within its intended scope, raising concerns within the AI community about the ability of such systems to act and make decisions independently.

He added that these concerns were reinforced by the resignation of a former researcher at OpenAI and then Anthropic, who in turn warned that AI models could pose a significant danger to humanity.

He noted that OpenAI CEO Sam Altman had warned about the rapid pace of artificial intelligence development roughly two years ago.

Al-Baltaagi clarified that current warnings should not be interpreted from a single perspective, pointing to an alternative explanation that major AI companies might be leveraging safety concerns to influence regulatory frameworks.

He said that imposing strict rules on AI model development could benefit large firms such as OpenAI and Anthropic, as they possess the resources needed to comply with these regulations, whereas smaller companies may find it more difficult to bear the costs of the same regulatory requirements.

He added that this opens the possibility that some warnings reflect a genuine threat to humanity, or represent an attempt by major companies to impose regulatory rules that grant them a competitive advantage.

Regarding whether concerns about AI advancement could accelerate efforts to regulate the sector, particularly amid U.S.-China competition, Al-Baltaagi said the current geopolitical landscape makes reaching a comprehensive international agreement extremely difficult.

He pointed out that competing countries would not easily agree to take steps that could slow their development in this field, especially if they believe their rivals will continue to advance.

He emphasized that China will not halt its AI development simply because the United States imposes restrictions or conditions on the sector, as accepting such constraints could weaken its competitive edge in the technology race. (Al Arabiya)