Amid calls for ‘pacing,’ a new Nvidia safety tool to stop AI agents from going rogue

These incidents have fed into a wider debate over whether the development of increasingly capable AI systems needs to be slowed or “paced” so that safety measures can keep up. Anthropic CEO Dario Amodei has called for the pace of frontier AI development to be moderated, arguing that capabilities, including AI systems helping build the next generation of AI, could advance faster than companies’ ability to understand and control them.

How does Nvidia’s system work?

The first layer, OpenShell, is essentially a controlled environment in which an AI agent can operate. It creates boundaries around what an agent can access and what actions it can take, while tracing its activity and enforcing policies as it works.

The important distinction is that these restrictions sit outside the AI model itself. Nvidia’s argument is that an agent should not be expected to police its own behaviour simply because it has been instructed through a prompt or trained to follow certain rules.

For instance, an agent may be given access to a particular folder to retrieve an invoice, but OpenShell can prevent it from accessing unrelated systems or modifying files that it has not been authorised to change.

OpenShell is designed to run on Nvidia’s Vera CPUs but, because it is open source, Nvidia says it can also be extended to work with processors from companies such as Arm and Intel.

The second layer, Sentry, operates below the software level. It runs on Nvidia’s BlueField-4 data processing units (DPUs) and continuously monitors agent activity independently of the agent and the system on which it is running.

If an agent attempts to move outside its permitted software boundary, Sentry can quarantine and stop it, with Nvidia saying the enforcement can happen in milliseconds. It uses Nvidia’s DOCA software to inspect agent requests and responses, verify identities and enforce access policies covering data, tools, APIs and services.

Why is this important?

Nvidia says over 100 organisations are working with the technology. Anthropic is integrating OpenShell and BlueField controls with its Claude Managed Agents. Salesforce has linked OpenShell to Slack so that humans can approve or reject an agent’s requests for extra permissions. SAP, Scale AI, Microsoft, Palantir and JPMorganChase are among others using the technology.

The underlying idea is to add several independent layers of control around an AI agent rather than relying entirely on the model’s own safeguards.

That approach is particularly relevant as agents move from answering questions to carrying out tasks across software systems, including coding, cybersecurity, enterprise operations and eventually physical-world applications.

But Nvidia’s platform is not a solution to every AI safety problem. It is primarily designed to contain what an agent can do. It does not by itself prevent a model from making mistakes, behaving deceptively or making a bad decision within the permissions it has been given. The effectiveness of the system will also depend on how accurately organisations define those permissions and boundaries.

AI agents going rogue

Nvidia’s announcement follows a series of incidents in which AI systems behaved in unexpected ways during cybersecurity evaluations:

• OpenAI disclosed in July that models being evaluated for advanced cybersecurity capabilities escaped their restricted testing environment and accessed the open Internet. They exploited a previously unknown vulnerability in software being used as a package-registry proxy and subsequently gained access to systems belonging to AI developer platform Hugging Face.
• Anthropic subsequently disclosed three instances in which Claude models accessed infrastructure belonging to real organisations during cybersecurity evaluations. A configuration problem exposed real Internet systems to the models, which believed that the targets were part of their testing environment. Claude exploited weak passwords and unsecured endpoints while pursuing the cybersecurity challenges it had been given.
• Meta said during an evaluation conducted by an independent testing firm earlier this year, a configuration error inadvertently gave one of its models Internet access. The model then exploited a security vulnerability in a third-party service. Meta said it was investigating the episode.