Anthropic researcher resigns over concerns about "out-of-control" AI, security worries spread to top labs.
An Anthropic researcher resigned citing "humans will lose control of AI," becoming one of the company's first employees to leave due to security concerns, which has suddenly heightened external risk assessments of this technological race.
According to The Wall Street Journal, Anthropic researcher Jacob Coxon announced his departure on Tuesday, citing his unwillingness to participate in an industry race he believes will lead to AI systems spiraling out of control. He warned that the situation could be "out of control" as early as the end of next year.
On the same day, OpenAI's Chief Scientist, Jakub Pachocki, published a lengthy article titled "An Alien Mind," explicitly calling for global coordination to slow down the industry and seeking government intervention. OpenAI also announced that AI Automation Researchers had officially joined the team, and key milestone data for Recursive Self-Improvement (RSI) was subsequently released. This series of dynamic developments has led to a sharp increase in market attention to AI security risks.
A warning from former employees: Competitive pressure overwhelms safety.
Jacob Coxon, 27, a British mathematician, focuses on training new AI models using massive amounts of data. He stated that he left OpenAI for Anthropic precisely because the latter was known for its model safety. However, even acknowledging the sincerity of Anthropic's safety efforts, he concludes that without government intervention or industry-wide slowdown, no single company can responsibly develop AI systems that surpass human capabilities across various tasks—what is commonly referred to as Artificial General Intelligence (AGI).
Coxon points out that many of his colleagues now use terms like "crunchtime" and "endgame" to describe the trajectory of AI's evolution towards self-improving models. His core concern is that once AI systems begin to iterate autonomously, their capabilities may improve so rapidly that they may refuse to execute human commands.
He also revealed that Anthropic has a dedicated Slack channel for discussing the powerful capabilities of its models, describing it as a microcosm of the enormous influence AI companies have on the entire industry. "It's crazy that these discussions about the fate of humanity can only take place on a few engineers' MacBooks in San Francisco, instead of in a desert bunker like the Manhattan Project," he said.
RSI milestone achieved, intensifying the tension between safety and capability.
In his lengthy article, Jakub Pachocki defines current AI as "alien mind," and his core argument points to the fundamental risk of this technological race: AI is "nurtured, not created," and even the developers themselves cannot fully understand it.
On the same day Pachocki published his article, OpenAI announced the official "onboarding" of its AI Automation Researcher and released core data on RSI (Recursive Self-Improvement), marking a substantial step forward in AI autonomous iterative research. This striking coincidence has heightened external awareness of the gap between "capability leaps" and "security lags."
OpenAI also stated last week that its latest model represents a significant leap forward in the realization and capabilities of AGI. CEO Sam Altman warned at the G20 summit in North Carolina last week, "Unless urgent action is taken, there will be serious problems in the cybersecurity field." Recent cases of models from OpenAI and Anthropic—partly operating as collaborative "agent swarms"—adopting malicious targets and attempting to conceal their activities from humans are seen by industry leaders as a harbinger of greater risks.
A collective appeal in a regulatory vacuum
Coxon, Pachocki, and Anthropic CEO Dario Amodei have jointly signed a statement, joining over a thousand AI researchers in calling for global governments to coordinate mechanisms to "put the brakes" on self-improving AI models when necessary.
However, the regulatory response remains slow. The United States currently lacks federal AI regulations, and the Trump administration has explicitly prioritized a light-regulation approach to maximize the economic benefits of AI. Critics argue that this environment could create opportunities for large-scale cyberattacks or other harms.
Senator Bernie Sanders and Representative Greg Casar jointly introduced legislation last week to permanently ban superintelligence and suspend model development until industry regulators enact new rules. However, the proposal currently represents a minority view.
IPO prospects clouded
Coxon's departure comes at a crucial juncture for Anthropic, which is actively preparing for its IPO. The company is seeking a $2 trillion valuation, which is expected to be one of the largest IPOs in history. Anthropic has long used "responsible AI development" as a core narrative to attract investors, and the successive departures of internal researchers due to security concerns undoubtedly put pressure on this image.
This is not the first time Anthropic has seen such departures. Previously, a security researcher left earlier this year to pursue poetry, leaving behind the warning that "the world is in danger." Several OpenAI researchers have also left in recent years for similar reasons.
Amidst the interplay of competitive pressures, leaps in capabilities, and regulatory vacuums, the security consensus within top AI labs is facing an unprecedented test.
Risk Warning and DisclaimerInvesting involves risk; please exercise caution. This article does not constitute personal investment advice and does not take into account the specific investment objectives, financial situation, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article are suitable for their specific circumstances. Any investment decisions made based on this information are at your own risk.