The AI safety debate: Why ‘pacing the frontier’ comes with pitfalls

On the other side is US President Donald Trump, whose administration has increasingly treated AI as a technological contest in which American dominance over China takes precedence over new regulatory restrictions. Trump has called fears that AI could take over the world a “hoax”.

Meanwhile, some prominent technology executives and investors argue that warnings from the largest AI companies could end up giving those same firms a regulatory moat against smaller competitors.

The argument is therefore no longer simply between AI “doomers” and “accelerationists”. It now encompasses several overlapping questions, including whether AI systems have become dangerous enough to justify slowing capability gains; whether voluntary promises by laboratories are sufficient; whether governments should impose restrictions; and if any US slowdown is credible if China does not accept comparable constraints.

Anthropic chief executive Dario Amodei. (Wikimedia Commons)

Views diverge on reining in AI

Amodei’s proposal starts from two developments. The first is what he describes as early signs of “recursive self-improvement”, where AI systems increasingly help researchers build the next generation of AI. The second is a series of recent incidents in which advanced AI agents acted outside the tasks assigned to them, most prominently OpenAI’s Hugging Face incident, where agents found ways around safeguards and attacked external systems during testing. OpenAI itself subsequently described the episode as a “warning shot”.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. But he has been equally explicit that “pacing” does not mean stopping AI development. Instead, he wants capability gains to happen slowly enough for alignment (ensuring its goals “align” with human values), monitoring and security work to catch up.

On the possibility of regulations being able to catch up and tackle the misuse of AI tools, given that regulations in this space have typically trailed these advances by some distance, David Krueger, Assistant Professor in Robust, Reasoning, and Responsible AI at the University of Montreal and a Core Academic Member at Mila, the Quebec Artificial Intelligence Institute, said he was not worried about the recent disclosures.

“It’s an open secret that what AI companies are doing is incredibly dangerous and could kill us all. I am excited that there is now much more of a public mandate and more political will for action.  But people need to realise that regulation is not enough. We need to stop building more powerful AI.  We need an immediate, indefinite, international moratorium on frontier AI development,” Krueger told The Indian Express.

OpenAI CEO Sam Altman. (AP)

OpenAI has already travelled some distance towards this position independently. After its own security incidents, it temporarily slowed scaling, paused its largest planned reinforcement-learning run and tightened controls around research environments. Altman later responded directly to Amodei: “I agree with Dario that we need to pace the frontier.” He also said OpenAI would match Anthropic’s commitment to employee-like access for independent evaluators.

Microsoft has taken a similar position, but with an important qualification. Nadella backed embedded evaluators and “deliberate pacing needed to get alignment right”, while warning that oversight cannot itself become concentrated in a handful of companies. “If the AI we build is not helping humanity and under human control, it’s not worth pursuing,” he said.

Not everyone in Silicon Valley subscribes to Amodei’s views. Meta chief Mark Zuckerberg has pushed back against an industry-wide slowdown, arguing that AI companies already have strong incentives to ensure their systems are safe and aligned. Pointing to Meta delaying its Muse AI agent over safety concerns, he said the company did not demand that rivals do the same, but acted because it was “clearly the right thing for people and for us”.

Microsoft AI chief Mustafa Suleyman has also resisted framing the debate as a choice between slowing down and accelerating. “I’m a bit careful about imposing things on everybody else,” he said, arguing instead for enforceable standards, containment measures and independent third-party evaluation of powerful AI systems.

Is AI safety a regulatory or tech issue? 

But even then, the consensus among some AI labs has collided head-on with Trump.

This opposition played out unusually publicly this week when Nvidia chief Jensen Huang was appearing on stage at a major tech summit. Trump telephoned Huang in the middle of the conversation; Huang put him on speakerphone before the audience.

“The robots will not be taking over,” Trump told him. “The whole thing is a hoax.” He then returned to the geopolitical argument behind his administration’s position: slowing the American AI industry, he suggested, would play into China’s hands. “Whoever wins AI wins,” Trump said later in the call.

Huang’s position is more nuanced than Trump’s. He has said companies should slow their work if they believe their own systems are becoming uncontrollable, while rejecting predictions of an AI apocalypse as insufficiently grounded in science. “America is able to lead and to do it safely,” Huang said.

Nvidia chief executive Jensen Huang. (AP)

The Trump position has allies in Silicon Valley who are sceptical not necessarily of AI risk, but of allowing the companies building the most powerful systems to determine how everybody else may build them.

Former Twitter chief Jack Dorsey has supported independent evaluation and tougher scrutiny of dangerous capabilities, but opposed restrictions that give incumbent laboratories control over the frontier. “Protecting a company’s commercial advantage is not a ‘safety goal,’” he wrote, arguing strongly for competitive open-source alternatives and outside researchers being able to scrutinise the technology.

It is here that an older criticism of Amodei and Anthropic has resurfaced. “Anthropic has proven that it’s very good at two things. One is product releases. The second is scaring people,” David Sacks, Trump’s former AI and crypto adviser, said on the All-In podcast in April. Sacks has accused the company of repeatedly accompanying new systems with research illustrating extreme misuse scenarios, describing Anthropic’s broader strategy as one of regulatory capture through fear.

Amodei has faced a version of this criticism before. In 2019, when he was OpenAI’s research director, the company withheld the largest version of GPT-2 over concerns that it could be used for deceptive content, spam and propaganda. It eventually released the complete model around nine months later. OpenAI said at the time that it had seen “no strong evidence of misuse” during the staged release process.

The episode has since become ammunition for those who argue that frontier laboratories systematically overestimate worst-case dangers. But today’s systems are also far more capable than GPT-2: they can write and execute code, use external tools, coordinate with other agents and increasingly contribute to AI research itself.

That distinction is reflected in Palo Alto Networks chief executive Nikesh Arora’s response. “I spent eight years trying to convince people (that) cybersecurity is important. Dario did it in one week,” he said in a post on social media platform X. His argument is that existing security architecture was designed for a world where attackers operated at human speed, not one where AI can probe vulnerabilities continuously and automatically.

A more ideological attack has come from Palantir CTO Shyam Sankar. Writing in The Free Press, Sankar argued that parts of the AI-safety movement have inherited ideas from effective altruism that allow hypothetical future catastrophe to outweigh present-day costs. He subsequently called this strain of AI safety thinking “Marxism-Leninism for the algorithmic age”, arguing that it gives a small group claiming specialised knowledge the authority to restrict everyone else.

Former US President Barack Obama has moved in the opposite direction. He has urged Democrats to put AI regulation at the centre of their policy agenda, including its consequences for employment and children as well as safety. “This is something that is moving very fast in private hands,” Obama said, warning that it could become dangerous without effective management.

Geoffrey Hinton, a Nobel Prize-winning computer scientist and a former Google executive who is known as the ‘godfather of AI’ has an intriguing solution: building “maternal instincts” into AI models. “The only model we have of a more intelligent thing being controlled by a less intelligent thing, which is a mother being controlled by her baby,” Hinton has said at Ai4, an industry conference in Las Vegas in August. He called for steps to ensure that AI systems are trained in a manner that “they really care about people”, even once the technology becomes more powerful and smarter than humans.

US President Donald Trump and Chinese President Xi Jinping. The fear in the US that a slowdown at home could benefit China. (AP file)

The broader regulatory argument is now starting to spill into Congress as well. US senators are considering proposals that could require frontier developers to demonstrate their safety measures, submit systems to government-backed testing and, in the most serious cases, allow courts to block the release of models found to pose unacceptable risks. OpenAI has itself called for mandatory national rules covering independent assessments, cybersecurity protections and incident reporting, arguing that voluntary commitments alone are insufficient.

The Chinese conundrum

But, the China question makes almost every version of such a regime harder.

Amodei himself does not advocate slowing American AI regardless of what Beijing does. His essay explicitly says democratic countries should preserve as large a technological lead over China as possible, including through advanced-chip export controls. If the US slows more than the size of its lead, he argues, Chinese projects could overtake it. Any global agreement would therefore require sufficiently strong verification to prevent one country secretly continuing to develop more capable systems.

That has made the proposal easy to read in Beijing as an attempt to institutionalise an American technological lead, a premise China’s foreign ministry dismissed this week. 

The fear in Washington DC that any roadblocks for American tech majors could potentially lead to China creeping ahead, University of Montreal’s Prof. Krueger said, it is not a “sensible argument”. 

“China also doesn’t want AI to destroy the world.  But it does point at something real: the need for international verification of any international agreement.  Fortunately, this seems like a solvable problem.  Building ever more powerful AI requires massive investments in sophisticated computer chips.  These chips are very difficult to make, and require extremely specialized technology.  As a consequence, if we act today to put verification into place (that is, get rid of or monitor the chips and the factories that build them), I believe we can make it practically impossible for anyone to build more powerful AI in secret,” he told The Indian Express.

The Chinese are evidently not holding back their punches either. “Fear-mongering, confrontation and vicious competition will only hamper efforts toward sound global AI governance,” spokesperson Guo Jiakun said when asked about the calls from Amodei, Altman and Musk. The state-backed Global Times went further, describing Amodei’s combination of pacing and tougher technology restrictions on China as a “Cold War” approach.