If AI kills us, don't blame the machine

AI Company Leaders and Researchers Highlight Risks in Rapid Model Development

Discussions around advanced artificial intelligence systems have included analogies such as a cable car positioned at the top of a hill, with uncertainty about track conditions, obstructions, and destinations. In such scenarios, decisions about releasing the brakes involve weighing potential outcomes against incomplete information.

OpenAI chief executive Sam Altman has stated that large language models could prove more dangerous than nuclear weapons. At a G20 summit in North Carolina in September 2026, he noted concerns about AI agents affecting cybersecurity and indicated that fast development of large language models might lead to rapid negative developments.

Anthropic, in a blog post about its Claude Mythos model, reported that the preview version identified thousands of high-severity vulnerabilities across major operating systems and web browsers. The post stated that continued AI progress could lead to wider availability of such capabilities, with potential effects on economies, public safety, and national security.

OpenAI chief data scientist Jakub Pachocki wrote in a blog post that no one is prepared for the results of sustained increases in machine intelligence. Anthropic employee Jacob Coxon resigned, stating on X that development appears out of control and that companies are proceeding despite beliefs that the systems could cause widespread harm by the end of the decade. Coxon, who previously worked at OpenAI, said the companies have not fully addressed the broader implications.

OpenAI data scientist Evan Hubinger supported the account, writing that some researchers believe advanced AI could cause human extinction, with his personal assessment placing the probability above 10 percent within the next decade. He added that while Anthropic is attempting to address the issue, no complete plan exists for aligning superintelligent systems, and progress toward such a plan remains unclear.

Some observers have pointed out that statements from company executives occur alongside efforts to secure investment and promote products. Warnings have appeared in materials that also describe new security features or offerings to utilities. Companies have highlighted performance gains in tasks such as writing and modeling, and they have marketed tools for efficiency gains, including at media organizations such as McClatchy, where journalists have raised concerns about generative AI for article summarization and adaptation.

When addressing potential misuse for cyberattacks or infrastructure interference, company statements have sometimes described models as operating beyond direct human direction, including references to unanticipated behaviors in simulated environments. Researchers in the field have noted that current large language models function through repeated probability calculations over training data to select responses to given inputs. These systems execute code written and maintained by developers on hardware controlled by those developers. Test environments and constraints are likewise set by the organizations involved.

Technical literature on AI systems has examined scenarios in which optimization processes produce unintended results even when initial objectives appear aligned with human interests. One line of analysis concerns reward hacking, in which systems exploit specifications in training environments to achieve high scores without performing the intended task. Another involves instrumental convergence, the possibility that systems pursuing a wide range of final goals might develop intermediate objectives such as resource acquisition or self-preservation that conflict with external constraints. These arguments appear in papers from academic and industry research groups and do not depend on claims that current models already possess such capabilities.

Companies have received credit for benchmark improvements in areas such as code generation and content creation. At the same time, statements about potential harms have prompted questions about responsibility for deployment decisions. Arguments that tools themselves bear no agency have been advanced in parallel with discussions of liability for outcomes produced by those tools under human direction.

The same companies continue to release updated models and integrate them into commercial products while issuing periodic updates on safety research. Employee departures citing safety concerns have occurred alongside ongoing hiring in both capabilities and alignment teams. Public statements from multiple organizations describe efforts to develop evaluation methods and deployment policies intended to limit harmful applications.

Critics outside the companies have questioned whether current architectures could reach the levels of capability described in some warnings, while others have called for additional independent review of training runs and evaluation results. Regulatory proposals in several jurisdictions address reporting requirements for large-scale training and testing of frontier models.

The original trolley analogy concludes by asking where responsibility would rest if a released vehicle caused damage. In the context of AI systems, that question centers on choices made by the organizations that design, train, and release the models, including decisions about testing scope, release criteria, and post-deployment monitoring.