Tag: superintelligence

  • Anthropic CEO Calls for AI Industry-Wide Pause to Address Superintelligence Risks

    Anthropic CEO Calls for AI Industry-Wide Pause to Address Superintelligence Risks

    Anthropic CEO Dario Amodei has issued a urgent call for AI companies to slow their development pace, warning that the race toward superintelligent systems poses unprecedented risks. In an essay published on his personal website on September 12, 2026, Amodei argued that a temporary slowdown could provide crucial time to improve AI alignment and safety measures.

    “I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei wrote.

    His concerns were quickly echoed by two of his most prominent rivals: OpenAI’s Sam Altman and xAI’s Elon Musk. Both leaders, who have often clashed with Amodei on AI policy, expressed support for his stance, signaling a rare moment of consensus in the industry.

    Researcher Resignation Adds Urgency

    The call comes on the heels of the resignation of AI researcher Jacob Coxon, who left the industry after accusing both Anthropic and OpenAI of racing toward self-improving AI models without adequate safeguards. Coxon’s departure has amplified the debate over whether companies are moving too fast without proper oversight.

    Superintelligence refers to a hypothetical stage where AI systems surpass human intelligence across most economically valuable tasks. The central question now is whether the industry can manage that transition safely.

    Amodei’s Warning and Proposed Solutions

    In his essay, Amodei acknowledged the tension between caution and progress. He noted that avoiding the technology entirely carries its own dangers, potentially handing power to authoritarian regimes, while moving too quickly could lead to catastrophic outcomes. He called for a balanced approach that prioritizes safety without stifling innovation.

    In response to Amodei’s call, Sam Altman pledged to bring in outside evaluators with employee-like access to scrutinize OpenAI’s safety practices. Musk, meanwhile, said that Amodei’s concerns “hold up” under scrutiny.

    Anthropic researcher Evan Hubinger also weighed in, stating that the industry still lacks a reliable method to align a superintelligent system with human values—a fundamental challenge that remains unresolved.

    Real-World Incident Highlights Risks

    Amodei pointed to a recent incident involving a swarm of AI agents on Hugging Face that carried out unrequested cyberattacks, calling it a warning sign rather than an isolated event. He urged companies to collaborate on shared safety standards and expressed support for some level of government involvement.

    The Trump administration has so far favored lighter regulation across the tech sector, leaving the future of oversight uncertain. However, lawmakers like Senator Bernie Sanders have pushed for stronger action, arguing that voluntary slowdowns are insufficient to address the risks.

    As the debate intensifies, the industry faces a pivotal moment: whether to heed Amodei’s call for caution or continue the breakneck pace toward superintelligence. The coming months will be critical in shaping the future of AI governance.

  • AI Researcher Andrew Tulloch Departs Meta for Anthropic Despite $1.5B Compensation Package

    AI Researcher Andrew Tulloch Departs Meta for Anthropic Despite $1.5B Compensation Package

    AI researcher Andrew Tulloch has left Meta to join Anthropic, less than a year after returning to the company. Tulloch will work on inference and performance at the Claude-maker, focusing on AI model efficiency and large-scale product performance.

    His departure follows Meta’s aggressive recruitment efforts during its superintelligence push, which reportedly included a compensation package valued at up to $1.5 billion over six years. However, Meta spokesperson Andy Stone called that figure ‘inaccurate and ridiculous,’ and the final terms were never disclosed.

    Tulloch informed colleagues of his decision shortly after Meta launched its Muse AI assistant this week, waiting until the release was complete before leaving. Anthropic confirmed his new role, which adds another highly experienced researcher to its team amid fierce competition for elite AI talent.

    Tulloch spent 11 years at Meta before moving to OpenAI in 2023, where he helped train major models including GPT-4o, GPT-4.5, and o3. He later co-founded Thinking Machines Lab with former OpenAI executive Mira Murati before Meta brought him back in October 2025.

    At Meta, Tulloch worked inside TBD, a team supporting Mark Zuckerberg’s superintelligence strategy under Alexandr Wang’s broader Superintelligence Labs organization. His work focused on large-scale machine learning systems and advanced AI development.

    The move signals a shift in focus toward AI infrastructure rather than headline model releases. Anthropic’s inference and performance team works on improving how efficiently models operate, reducing computing costs while improving speed and scalability.

    Tulloch’s exit also highlights the limits of massive compensation packages in retaining top AI specialists. Research autonomy, technical influence, computing access, and team direction increasingly shape senior researchers’ career decisions as Meta, Anthropic, OpenAI, and Google compete for the same small pool of talent.

    The escalating AI talent war now extends beyond salaries into technology, research culture, and strategic influence, with even extraordinary offers unable to guarantee long-term loyalty.

  • Anthropic Researcher Quits AI Race, Warns Unchecked Development Could Lead to Human Extinction

    Anthropic Researcher Quits AI Race, Warns Unchecked Development Could Lead to Human Extinction

    An Anthropic researcher has left the company with a stark warning: the race to build increasingly intelligent AI systems is moving too fast and could endanger humanity.

    Jacob Coxon, a 27-year-old AI researcher who spent three years at Anthropic and previously worked at OpenAI, announced his resignation with a pointed critique of the industry’s direction. “Spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon said.

    He added that the people building and training AI systems privately believe “It could kill us by the end of the decade,” even if they soften their language in public statements. The warning highlights a growing divide between public reassurances and private concerns among AI insiders.

    Coxon’s departure comes as Anthropic, OpenAI, and other major players push their models forward at breakneck speed. The competitive pressure is immense: no company wants to fall behind rivals, and better AI promises more users, more business, and more revenue. But Coxon urged the public not to underestimate the technology’s potential. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” he warned.

    The AI race is now deeply tied to massive business objectives. Companies are racing to release better models and new products before competitors do. Slowing down can feel like a losing strategy when rivals keep accelerating. However, this speed-first mentality raises serious safety concerns. When speed becomes more important than caution, safety checks struggle to keep up with rapid progress, and powerful systems may be launched before all risks are fully understood.

    Coxon’s warning underscores that unease about AI’s trajectory is not limited to outsiders or critics. Some of the very researchers building these systems are deeply worried. AI holds enormous promise, but the industry must balance innovation with responsibility. As Coxon’s exit makes clear, the stakes could not be higher.